Image processing device and image processing method

ABSTRACT

An image processing apparatus for automatically improving the contrast of an input image that is obtained from a digital camera or the like, and obtaining a sharper and clearing image. A contrast improvement unit ( 11 ) performs a contrast improvement process on the input image by comparing an object pixel in the input image with pixels in the surrounding area. An image combination unit ( 12 ) combines the enhanced image obtained by the contrast improvement process with the input image. The combined image is then output to a desired device such as a printer by an image output unit ( 13 ).

TECHNICAL FIELD

This invention relates to an image processing apparatus and imageprocessing method for automatically improving the contrast of an imageobtained using a digital camera or the like.

BACKGROUND ART

The dynamic range of a Digital Still Camera (DSC) is greatly limited bySN levels which indicate the noise ratio of analog values obtained fromthe CCD image sensor and by the precision of converting analog values todigital values. Therefore, detail in dark area tends to be lost in theimage taken by the DSC. Particularly, this tendency is large in sampleswhere there are both light areas and dark areas.

As a method of improving the quality of the image, first there is amethod of enhancing the contrast such that the brightness range of thedigital image extends from the region of high brightness to the regionof low brightness.

As a conventional method of enhancing the contrast there is the methodof histogram equalization. This method is a method that creates ahistogram showing the distribution of brightness of all of the pixels onthe original image, and uses the accumulated curves of that histogram asthe brightness conversion curve to convert the brightness values of thepixels of the original image to new values.

In this method, in order to use the same brightness conversion curve toconvert the brightness value of every pixel in the whole of the originalimage to a new brightness value, there may be a partial decrease incontrast.

In order to avoid this, a contrast enhancement process that is suitablefor a part can be performed. Many procedures to do this have beenproposed such as a localized histogram equalization method that dividesthe image into a plurality of rectangular areas and performs histogramequalization for each of the areas.

For example, as disclosed in Japanese unexamined patent publication No.2000-285230 and as shown in FIG. 1, a contrast correction unit comprisesan image division unit 2001, histogram creation unit 2002 and contraststretching unit 2003.

The image division unit 2001 divides the input image into rectangles.The histogram creation unit 2002 creates histograms for each of therectangles. The contrast stretching unit 2003 performs contraststretching for each of the rectangles.

In the method of this disclosure as well, problems occur in that thereare rectangular areas for which the contrast is enhanced too much, andthe contrast is not continuous at the boundary between adjacentrectangular areas.

A technique has also been proposed that solves these problems in which ahistogram is not used. For example, as disclosed in Japanese unexaminedpatent publication No. H6-141229, the shutter time and lens stops of thedigital camera are changed for each field, to photograph the light areasand dark areas separately. By combining both of the obtained images intoone image, halftone densities are presented. This makes it possible toobtain an image having a large dynamic range.

FIG. 2 is a block diagram showing the construction of the imageprocessing apparatus disclosed in Japanese unexamined patent publicationNo. H6-141229. In this image processing apparatus, the image sensor 2101performs photoelectric conversion of the light image of the photographedobject. An image combination unit 2102 weights and combines two or moreimages having different electric charge storage periods in the imagesensor according to the signal levels of the images. In order toaccomplish this, the image combination unit 2102 comprises: a memory2103, a multiplication unit 2104, level weighting units 2105 and 2106,and an adding unit 2107. The memory 2103 stores image signals. Themultiplication unit 2104 multiplies a constant to a signal level. Thelevel weighting units 2105 and 2106 modify the image signals by weights,and the adding unit 2107 adds the image signals.

Also, a speed conversion unit 2108 converts the speed of the imagesignal, and a level compression unit 21 9 compresses the level of theimage signal. Moreover, a timing control unit 2110 controls the timingof each block. This apparatus is for a television imaging system thatcompresses a television signal to a standard level, so in order toconvert the obtained combined image output to speed of a standardtelevision signal, there is a speed conversion unit and a levelcompression unit. In the case of applying this kind of technique to adigital camera, the speed conversion unit and level compression unit arenot necessary.

In the case of the method of combining images obtained for a pluralityof electric charge storage periods as described above, it is unlikelythat a discontinuity of contrast occurs in the combined image. However,since this method is required to take at least two images in sequence,this is impossible to take the same images in principal. Therefore, whenthe images are combined, there is a possibility that an image will becreated in which the detailed parts of the combined image will beblurred or shifted, although depending on the shutter speed. Also, whenit is not possible to cover the entire density range of the image withthe density range obtained when photographing the light area and thedensity range obtained when photographing the dark area, there is adanger that discontinuity will occur in the middle density range.

Also, a method for improving the image quality of a digital image byusing the Retinex theory has been disclosed in International PublicationNo. WO97/45809 or Published Japanese translation of PCT internationalapplication No. 2000-511315. When a person observes an object, theproblems mentioned above do not occur for detailed areas and colors indark areas. People are capable of visually perceiving the large densitydynamics and colors of the original images. Taking notice of this humanvisual perception, the concept of center/surround Retinex was introducedby Edwin Land in the publication ‘An Alternative Technique for theComputation of the Designator in the Retinex Theory of Color Vision’,National Academy of Science, Vol. 84, pp. 3078 to 3080 (1986).

This document explains the concept of the Retinex theory and states thatin human visual perception, the center view can be represented by aninverse square function having a diameter of from two to four basicunits, and that the surround view can be represented by an inversesquare function having a diameter of approximately 200 to 250 times thatof the center view. Also, the spatial average of the signal intensity inthe field of view of both the center view and surround view is definedas being related to the perceived intensity. The method described in thepamphlet or publication is one method of improving the expression ofcolor and lightness in dark areas according to this kind of theory.

FIG. 3 is a block diagram that explains the image processing apparatusdescribed in the pamphlet or publication. Here, the image processingapparatus is explained for a grayscale image as an example, however,that explanation can be expanded to include color images as well.

The processor 2202 and filter 2203 perform adjustment and a filteringprocess on the values I (i, j) of the pixels (i, j) of the imageobtained from the digital image pickup apparatus 2201.

The processor 2202 calculates the adjusted values I′(i, j) given by thefollowing equation 1 for each pixel.I′(i,j)=logI(i,j)−log[I(i,j)*F(i,j)]  (Eq. 1)

Here, F(x, y) is the surround view function that expresses the surroundview, and ‘*’ denotes the convolution operation. By setting thenormalization coefficient K such that the condition of Equation 2 belowis satisfied, the second term of Equation 1 is the weighted averagevalue of the pixel values of the area corresponding to the surroundview.K∫∫F(i,j)di dj=1  (Eq. 2)

In other words, Equation 1 is an equation that corresponds to thelogarithmic conversion of the ratio of each pixel to the average valuein a large area. The surround view function F(i, j) is designed from thecorrespondence to the model of human visual perception such that thecontributing ratio becomes higher closer to the object pixel, and aGaussian function such as that of Equation 3 below is applied.F(i,j)=exp(−r ² /c ²)r=(i ² +j ²)^(1/2)  (Eq. 3)

Here, c is a constant for controlling the adjusted values I′(i, j) foreach of the pixel values I(i, j).

When the ratio of the object to with the average pixel value of the areacorresponding to the surround view is calculated as the adjusted valueI′(i, j), the filter 2203 performs a filtering process on this value,and generates Retinex output R(i, j). This filtering process is aprocess of converting the adjusted value I′(i, j) in the logarithmicdomain to the value R(i, j) in the domain used by the display 2204, andto simplify this process, a process that uses the same offset and gainconversion function for all of the pixels is adopted.

One problem with this method is that the effect by the constant c thatcontrols the surround function is large. For example, when this constantc is a large value and the area corresponding to the surround view thatcontributes to improvement becomes large, then it is only possible tocompensate for colors in large shadows. On the other hand, when thisconstant c is a small value and only the pixel values near the objectpixel affects improvement, then the improvement is limited to acorresponding small area. Therefore, it is necessary to consider aconstant c that is suitable to the image being processed. In order toease this dependence, the document proposes a method in which areassuitable for a plurality of sizes of surround views are prepared.However, it is not clear how many area sizes should be prepared. Bypreparing many large areas and small areas in order to increase theimprovement precision, the processing time becomes very long.

Also, there is a problem in that knowledge from experience is necessaryin order to set a suitable offset and gain conversion function.

Furthermore, when there is very small change in the pixel value in thelargest area of the areas set by the plurality of constants c, theadjusted value I′(i, j) comes close to 1.0 regardless of the value I(i,j) even when a plurality of areas are prepared. An adjusted value I′(i,j) for an object pixel in the area with little change is often locatednear the average value of the adjusted values I′(i, j) for the entireinput image. Regardless of the offset and gain conversion function, theadjusted values I′(i, j) gather near the center of the histogram.Therefore, particularly in a large area uniformly having highlightbrightness, it becomes easy for the brightness to be adjusted in adirection that lowers the brightness and thus worsens the visualappearance. Also, in a large area where the brightness is low, forexample in a night scene, color noise and compression noise that occurwhen taking the photo appear due to excessive enhancement.

DISCLOSURE OF INVENTION

Taking the problems with the prior art into consideration, the object ofthis invention is to provide an image processing apparatus that caneasily improve the contrast with high quality.

In order to accomplish this objective, with the image processingapparatus of this invention, a contrast improvement unit performs acontrast improvement process on an input image by comparing an objectpixel of the input image with pixels that belong to areas surroundingthe object pixel. An image combination unit combines the enhanced imageobtained from the contrast improvement unit with the input image. Theimage after being combined is then output to a device such as a printerby the image output unit.

In this way it is possible to easily improve the contrast and improvethe image quality of the output image.

In the contrast improvement unit of this image processing apparatus, acorrection data calculation unit may find the amount to improve thecontrast for the pixels in the input image. An extraction unit extractsan effective range from distribution of the amount of contrastimprovement. A pixel value conversion unit converts the amount ofcontrast improvement of the object pixel to the value of thecorresponding pixel in the enhanced image according to extracted range.

This correction data calculation unit can also find the amount ofcontrast improvement by comparing the object pixel with pixels thatbelong to each of surrounding areas of different sizes. In this case, itis possible to reduce the effect due to the setting of constantsindicating the effect of the input image and the size of the surroundingareas.

Also, in the pixel value conversion unit, an average brightnesscalculation unit calculates the average brightness of the pixels in theinput image. A conversion method selection unit selects the method forconverting the contrast improvement amount to the pixel value in theenhanced image based on the average brightness. A pixel estimation unitconverts the contrast improvement amount to the pixel value of theenhanced image by the selected conversion method.

In this way it is possible to easily and accurately perform contrastimprovement.

The pixel value conversion unit may comprise a standard intensitycalculation unit, conversion curve estimation unit and pixel valueestimation unit. The standard intensity calculation calculates thestandard intensity value that indicates the intensity of the contrast ofthe input image. The conversion curve estimation unit estimates aconversion curve for converting the contrast improvement amount to thevalue of the enhanced image based on the standard intensity value. Thepixel value estimation unit uses the conversion curve to convert thecontrast improvement amount to the value of the enhanced image.

In this case, it is possible to easily improve the contrast by usingautomatically estimated conversion curves.

In this conversion curve estimation unit, an initial candidate settingunit may set an initial population of search vectors that express theconversion curves. A pixel value conversion candidate calculation unituses the candidates for the conversion curve that correspond to eachsearch vector to find a conversion value in the candidate for theenhanced image from the contrast improvement amount. An evaluation valuecalculation unit uses the standard intensity value and conversion valueto calculate an evaluation value for evaluating the candidates of eachconversion curve. A fitness calculation unit calculates the fitness ofthe candidates of the conversion curve based on the evaluation value. Arecombination operation unit performs recombination operation on theselected search vector based on the fitness for the candidates of eachconversion curve and generates a next generation population. Anestimation end judgment unit determines whether or not to end theestimation of conversion curves with the generation.

By performing estimation using a genetic algorithm in this way, it ispossible to automatically obtain an optimized conversion curve and toeasily perform contrast improvement with high quality.

It is also possible to find the amount of contrast improvement forsignals such the lightness or brightness signal. In that case, in thecontrast improvement unit, a signal conversion unit converts the pixelvalues of the input image to a plurality of signals that include thesignals that are the object of contrast improvement. An objectcorrection data calculation unit finds the amount of contrastimprovement of the object pixel for the object signal obtained from thesignal conversion unit. The extraction unit extracts an effective rangefrom the distribution of the contrast improvement amount for the objectsignal. An object signal conversion unit converts the contrastimprovement amount for the object signal to the value of the objectsignal in the enhanced image according to the extracted range. Theobject signal conversion unit finds the value of the pixels on theenhanced image based on the object signal of the enhanced image andsignals other than the object signal obtained by the signal conversionunit.

This correction data calculation unit can also find the amount ofcontrast improvement by comparing the value of the object signal of theobject pixel with the values of the object signals of pixels that belongto each of surrounding areas having different sizes. In this way it ispossible to perform high quality contrast improvement.

The object signal conversion unit may comprise an average object signalcalculation unit, and average object signal calculation unit and objectsignal estimation unit. The average object signal calculation unitcalculates the average value of the object signal of the input image.The average object signal calculation unit selects a conversion methodfor converting the contrast improvement amount of the object signal tothe value of the object signal of the enhanced image based on theaverage value. The object signal estimation unit converts the contrastimprovement amount of the object signal to a value of the object signalof the enhanced image using the selected conversion method.

In this way it is possible to easily and accurately improve thecontrast.

Furthermore, the object signal conversion unit may comprise a standardintensity calculation unit, an object signal conversion curve estimationunit and object signal estimation unit. The standard intensitycalculation unit calculates the standard intensity value that indicatesthe intensity of the contrast of the input image for the object signalobtained by the signal conversion unit. The object signal conversioncurve estimation unit estimates a conversion curve for converting thecontrast improvement amount of the object signal to the value of theenhanced image based on the standard intensity value. The object signalestimation unit uses the estimated conversion curve to convert thecontrast improvement amount to the value for the enhance image.

Even when performing processing for the object signal in this way, ageneric algorithm may be used for estimating the conversion curves. Inthat case, in the object signal conversion curve estimation unit, theinitial candidate setting unit sets an initial population of searchvectors that express the conversion curves. The object signal conversioncandidate calculation unit uses the candidates of the conversion curvethat corresponds to each of the search vectors to find a conversionvalue for the object signal in the candidate of the enhanced image fromthe contrast improvement amount for the object signal. The evaluationvalue calculation unit uses the standard intensity and conversion valueto calculate an evaluation value for evaluating the candidates of eachconversion curve. The fitness calculation unit calculates the fitnessfor the candidates of each conversion curve based on that evaluationvalue. The recombination operation unit performs recombination operationon the selected search vectors based on the fitness for the candidatesof each conversion curve and generates the next generation population.The estimation end judgment unit determines whether or not to end theestimation of conversion curves with the generation.

Also, in the image combination unit of this image processing apparatus,a selection standard value judgment unit may determine whether givepriority to the input image or enhanced image. A combination coefficientcalculation unit sets combination coefficients for the input image andenhanced image based on the result of the selection standard valuejudgment unit. A weighted average combination unit uses the combinationcoefficients set for respective images to generate a weighted averageimage for the input image and enhanced image.

By controlling the combination coefficients, it becomes possible toperform high quality improvement.

Also, in another image processing apparatus, the contrast improvementunit performs a contrast improvement process on the input image bycomparing the object pixel of the input image with the pixels in thearea surrounding the object pixel. An edge data detection unit extractsedge data of the input image. An image combination unit combines theenhanced image obtained from the contrast improvement unit with theenhanced image based on the edge data obtained by the edge datadetection unit. The image after combination is output by an image outputunit.

In this way it is possible to improve the contrast while suppressingdecreases in the density level of uniform highlighted areas and suddenincreases in the density level of shadow areas. Furthermore, it ispossible to decrease color cast.

In the contrast improvement unit of this image processing apparatus, thecorrection data calculation unit may find the amount to improve thecontrast for the pixels in the input image. The extraction unit extractsan effective range from the distribution of the contrast improvementamount. The pixel value conversion unit converts the contrastimprovement amount of the object pixel to the value of the correspondingpixel in the enhanced image according to the extracted range.

This correction data calculation unit can also find the amount toimprove the contrast by comparing the object pixel with pixels in eachof surrounding areas of different sizes. In this case, it is possible toautomatically improve the contrast of the input image without beingaffected by the dynamic range of the pixel value of the input image orthe size of dark areas such as shadows.

Also, in the image combination unit, the combination coefficientcalculation unit calculates the combination coefficients for the inputimage and the enhanced image based on the edge data obtained from theinput image. The weighted average combination unit generates a weightedaverage image for the input image and the enhanced image based on thecombination coefficients calculated for the respective images.

By controlling the combination coefficients based on the edge data, itis possible to improve the contrast while suppressing decreases in thedensity level of uniform highlighted areas and sudden increases in thedensity level of shadow areas. Furthermore, it is possible to decreasecolor cast.

Also, in yet another image processing apparatus, the contrastimprovement unit performs a contrast improvement process on the inputimage by comparing an object pixel of the input image with pixels thatbelong to an area surrounding the object pixel. A density correctionunit corrects the density distribution of the enhanced image obtainedfrom the contrast improvement unit according to the density distributionof the input image. The image combination unit combines the correctedimage obtained from the density correction unit with the input image.The image after combination is output by the image output unit.

In this apparatus, by correcting the density of areas where there arelarge changes, such as decreases in density that occur in highlightedareas or sudden increases in density in shadow areas of the enhancedimage, it is possible to match the density in the output image somewhatwith the input image.

In this case, in the image combination unit, the weighted averagecombination unit generates a weighted average image for the input imageand corrected image, and the output value setting unit sets the pixelvalues in the output image based on the image obtained by the weightedaverage combination unit and the input image.

Since the density is corrected, it is possible to suppress decreases indensity in highlighted areas and sudden increases in density in shadowareas in the image obtained from the weighted average combination unitas well.

Also, in yet another image processing apparatus, an edge data detectionunit detects edge data of the input image. The contrast improvement unitperforms a contrast improvement process on the input image bydetermining the area where the object pixel belongs based on the edgedata of the object pixel obtained from the edge data detection unit andthe brightness of the object pixel, and by comparing the object pixelwith the pixels that belongs to the area surrounding the object pixel.The image combination unit combines the enhanced image obtained by thecontrast improvement unit with the input image. The image aftercombination is output by the image output unit.

By determining the area of the object pixel based on the edge data, itis possible to sharpen the edges, and it is possible to improve thecontrast while suppressing the enhancement of noise in uniform shadowareas and decreases in density in highlighted areas.

In the image combination unit of this image processing apparatus aswell, the weighted average combination unit generates a weighted averageimage for the input image and corrected image, and the output valuesetting unit sets the pixel values in the output image based on theimages obtained by the weighted average combination unit and the inputimage.

Also, in yet another image processing apparatus, the edge data detectionunit detects edge data of the input image. The contrast improvement unitperforms a contrast improvement process on the input image bydetermining the area of the object pixel based on the edge data of theobject pixel obtained from the edge data detection unit and thebrightness of the object pixel, and by comparing the object pixel withthe pixels that belongs to the area surrounding the object pixel. Theimage combination unit combines the enhanced image obtained by thecontrast improvement unit and the input image. The image aftercombination is output by the image output unit.

By using the edge data in this way, it is possible to sharpen the edgesand it is possible to improve the contrast while suppressing theenhancement of noise in uniform shadow areas and decreases in density inhighlighted areas. Furthermore, by performing adaptive combination basedon the edge data, it is possible decrease color cast.

In the contrast improvement unit of this image processing apparatus, anarea judgment unit may determine the area to which the object pixelbelongs based on the edge data. A comparison range setting unit selectsa pixel comparison range based on the area obtained by the area judgmentunit. The correction data calculation unit finds the amount to improvethe contrast of the object pixel based on the pixel comparison rangeselected by the comparison range setting unit. An adjustment coefficientcalculation unit calculates an adjustment coefficient for the amount ofcontrast improvement based on the area obtained by the area judgmentunit. An adjustment unit corrects the amount of contrast improvementusing the adjustment coefficient obtained by the adjustment coefficientcalculation unit. The extraction unit extracts an effective range fromthe distribution of the corrected contrast improvement amount. The pixelvalue conversion unit converts the contrast improvement amount of theobject pixel to a value of the corresponding pixel in the enhancedimage.

This correction data calculation unit can also find the amount toimprove the contrast by comparing the object pixel with the pixels thatbelongs to each of surrounding areas of different sizes. In this way itis possible to suppress the bad influence caused by misjudgment on thesize of the area. Furthermore it is possible to decrease color cast onthe edges.

Also, in yet another image processing apparatus, the contrastimprovement unit has a correction data calculated density binding unitthat binds the density of the pixels that belongs to the areasurrounding the object pixel of the input image. In the bound state, thecontrast improvement unit performs a contrast improvement process on theinput image by comparing the object pixel of the input image with thepixels in the surrounding area. The image output unit outputs theenhanced image obtained by the contrast improvement unit.

In this way it is possible to improve the contrast while suppressingdecreases in the density of uniformly highlighted areas and suddenincreases in the density of large uniform shadow areas. Furthermore,since the enhanced image is output, it is possible to decrease colorcast.

This image processing apparatus may further comprise the imagecombination unit. This makes it to suppress decreases in density oflarge uniformly highlighted areas and sudden increases in density oflarge uniform shadow areas.

In the contrast improvement unit of this image processing apparatus, thecorrection data calculation unit may find the amount to improve thecontrast of the object pixel. The extraction unit extracts an effectiverange from the contrast distribution. The pixel value conversion unitconverts the contrast improvement amount of the object pixel to thevalue of the corresponding pixel in the enhanced image according to theextracted range.

This correction data calculation unit can also find the amount toimprove the contrast by comparing the object pixel with pixels thatbelongs to each of surrounding areas of different sizes. In this case,it is possible to automatically improve the contrast of the input imagewithout being affected by the dynamic range of the pixel values or thesize of the dark areas of the input image.

Also, in yet another image processing apparatus, a pre-processing unitperforms pre-processing on the input image. The contrast improvementunit performs a contrast improvement process on the pre-processed imageby comparing the object pixel of the pre-processed image with the pixelsin the surrounding area of the object pixel. The image combination unitcombines the enhanced image obtained by the contrast improvement unitwith the input image. A post-processing unit performs post-processing onthe combined image. The post-processed image is output by the imageoutput unit.

In this way, it is possible to properly improve the contrast whilesuppressing decreases in the density of uniformly highlighted areas andsudden increases in the density of shadow areas, even for an input imagefor which gamma conversion has been performed beforehand by a digitalcamera or the like.

In the contrast improvement unit of this image processing apparatus, thecomparison pixel setting unit may set comparison pixels to be used forcomparison from among the pixels in the surrounding area of the objectpixel. The correction data calculation unit finds the amount to improvethe contrast of the object pixel. A conversion standard valuecalculation unit finds a conversion standard value for converting thecontrast improvement amount to the value of the pixel in the enhancedimage. The pixel value conversion unit converts the contrast improvementamount to the value of the pixel in the enhanced image based on thatconversion standard value.

By finding the value of the pixel in the enhanced image based on theconversion standard value, it is possible to easily improve the contrastwhile suppressing flatness of the image that occurs in the combinedimage of the input image and enhanced image, and decreases in thedensity differences along the contour areas.

In this image processing apparatus, for example, the correction datacalculation unit comprises a surrounding average unit that finds theweighted average of the density of the comparison pixels, and animprovement amount calculation unit that finds the amount to improve thecontrast from the average density obtained by the surrounding averageunit and the density of the object pixel.

The correction data calculation unit may comprise a surrounding averageunit, edge data detection unit, correction coefficient calculation unit,comparison amount correction unit and improvement amount calculationunit. The surrounding average unit finds the weighted average of thedensity of the comparison pixels. The edge data detection unit detectsthe edge data of the object pixel. The correction coefficientcalculation unit calculates the correction coefficient for the edge dataobtained by the edge data detection unit. The comparison amountcorrection unit corrects the average density obtained by the surroundingaverage unit using the correction coefficient. The improvement amountcalculation unit finds the amount to improve the contrast from thecorrected average density and the density of the object pixel.

By controlling the average density in this way, it is possible toenhance the difference between the density of the object pixel and theaverage density, and it is possible to improve the flatness of the imagethat occurs in the combined image of the input image and enhanced image,and the decrease in the density difference in the contour areas.

Also, the correction data calculation unit may comprise a surroundingaverage unit, an improvement amount calculation unit, an enhancedcomponent calculation unit and improvement amount correction unit. Thesurrounding average unit finds the weighted average of the density ofthe comparison pixels. The improvement amount calculation unit finds theamount to improve the contrast from the average density obtained by thesurrounding average unit and the density of the object pixel. Theenhanced component calculation unit calculates the enhanced componentfrom the density difference of the comparison pixels and the objectpixel. The improvement correction unit adds the enhanced component tothe contrast improvement amount.

By doing this, as well, it is possible to improve the flatness of theimage that occurs in the combined image of the input image and enhancedimage, and the decrease in the density difference in the contour areas.

Also, in this image processing apparatus, processing can be simplifiedby using part of the pixels in the area surrounding the object pixel forcomparison with the object pixel. In that case, in the contrastimprovement unit, the comparison pixel setting unit sets the position inthe vertical direction of the pixel to be used in the comparison fromamong the pixels in the area surrounding the object pixel. A verticaldirection addition unit adds a weight in the vertical direction to thedensity of the comparison pixel obtained from the setting. A simplifiedsurrounding average unit calculates the comparison density for theobject pixel from the value at each horizontal pixel position in thesurrounding area obtained by the vertical direction addition unit. Theimprovement amount calculation unit finds the amount to improve thecontrast from the comparison density and the density and the density ofthe object pixel. The conversion standard value calculation unit finds aconversion standard value for converting that contrast improvementamount to the value of the pixel in the enhanced image. The pixel valueconversion unit converts the contrast improvement amount to the value ofthe pixel in the enhanced image based on the conversion standard value.

By cutting down the number of pixels in the vertical direction of thesurrounding area as well as in the horizontal direction, it is possibleto further simplify the process. In that case, in the contrastimprovement unit, the comparison pixel setting unit sets the positionsin the vertical direction and the positions in the horizontal directionof the pixel to be used in the comparison from among the pixels in thearea surrounding the object pixel. The selected vertical directionaddition unit adds a weighting to the vertical direction of the densityof the comparison pixels obtained from the setting. The simplifiedsurrounding average unit calculates the comparison density for theobject pixel from the addition value obtained by the selected verticaldirection addition calculation unit. The improvement amount calculationunit finds the amount to improve the contrast from that comparisondensity and the density of the object pixel. The conversion standardvalue calculation unit finds the conversion standard value forconverting that contrast improvement value to the value of the pixel inthe enhanced image. The pixel value conversion unit converts thatcontrast improvement value to the value of pixel in the enhanced imagebased on the conversion standard value.

Also, in this image processing apparatus, the pre-processing unitperforms inverse conversion of the gamma conversion that is preset inthe input image, for example, and the post-processing unit performs thegamma conversion.

Furthermore, by using a value that is proportional to the colorcomponent of the input image, it is possible to suppress colordistortion in the combined image, and thus it is possible to finallyoutput an image with high image quality. In that case, in thepost-processing unit an input brightness/color calculation unitcalculates the brightness value and color difference components of theinput image. A brightness adjustment unit compares the brightnesscomponent of the input image obtained by the input brightness/colorcalculation unit with the brightness component of the combined image,and adjusts the brightness component of the combined image. A colorcomponent correction unit corrects the color components of the inputimage obtained by the input brightness/color calculation unit based onthe brightness component of the combined image obtained by thebrightness adjustment unit. An image regeneration unit regenerates thecombined image using the brightness component of the combined imageobtained by the brightness adjustment unit and the corrected colorcomponents obtained by the color component correction unit. A gammaconversion unit performs gamma conversion on the combined image obtainedby the image regeneration unit.

Also, from another aspect, this invention provides an image processingmethod and image processing program that corresponds to this kind ofimage processing apparatus. That image processing program may beprovided over a telecommunication network such as the Internet, or maybe provided in a recorded state on a computer readable recording mediumsuch as a CD-ROM.

The image processing apparatus and image processing method will beexplained in detail with reference to the drawings as follows.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a prior image processing apparatususing a method of localized equalization.

FIG. 2 is a block diagram showing a prior image processing apparatususing a method for combining the image obtained in a plurality of chargestorage periods.

FIG. 3 is a block diagram that explains a prior image processingapparatus using the Retinex theory.

FIG. 4 is a block diagram showing the basic construction of the imageprocessing apparatus of a first embodiment of the invention.

FIG. 5 is a block diagram showing the contrast improvement unit of thefirst embodiment.

FIG. 6 is a block diagram showing the image combination unit of thefirst embodiment.

FIG. 7 is a diagram graphically showing human visual perception.

FIG. 8 is a flowchart explaining the contrast improvement process of thefirst embodiment.

FIG. 9 is a flowchart explaining the image combination process of thefirst embodiment.

FIG. 10 is a block diagram showing the contrast improvement unit of asecond embodiment of the invention.

FIG. 11 is a diagram graphically showing the surround view area.

FIG. 12 is a flowchart explaining the contrast improvement process ofthe second embodiment.

FIG. 13 is a block diagram showing the contrast improvement unit of athird embodiment of the invention.

FIG. 14 is a flowchart explaining the contrast improvement process ofthe third embodiment.

FIG. 15 is a block diagram showing the contrast improvement unit of afourth embodiment of the invention.

FIG. 16 is a flowchart explaining the contrast improvement process ofthe fourth embodiment.

FIG. 17 is a block diagram showing the pixel value conversion unit of afifth embodiment of the invention.

FIG. 18 is a flowchart explaining the pixel value conversion process ofthe fifth embodiment.

FIG. 19 is a block diagram showing the object signal conversion unit ofa sixth embodiment of the invention.

FIG. 20 is a block diagram showing the pixel value conversion unit of aseventh embodiment of the invention.

FIG. 21 is a block diagram showing the conversion curve estimation unitof the seventh embodiment.

FIG. 22 is a flowchart explaining the contrast improvement process ofthe seventh embodiment.

FIG. 23 is a flowchart explaining the conversion curve estimationprocess of the seventh embodiment.

FIGS. 24A and 24B are schematic diagrams that graphically show themarker chromosome structure used in the genetic algorithm.

FIGS. 25A and 25B are schematic diagrams that graphically show therecombination operation process of the genetic algorithm.

FIG. 26 is a block diagram shown the object signal conversion unit of aneighth embodiment of the invention.

FIG. 27 is a block diagram showing the object signal conversion curveestimation unit of the eighth embodiment.

FIG. 28 is a block diagram showing an image processing apparatus thatperforms auto white balance.

FIG. 29 is a block diagram showing the overall construction of the imageprocessing apparatus of a ninth embodiment of the invention.

FIG. 30 is a block diagram showing the construction of the imagecombination unit of the ninth embodiment.

FIG. 31 is a flowchart explaining the edge data calculation process ofthe ninth embodiment.

FIG. 32 is a diagram showing an example of the filter coefficient.

FIG. 33 is a flowchart explaining the image combination process of theninth embodiment.

FIG. 34 is a concept diagram showing the relationship between the edgedata and image combination process.

FIG. 35 is a concept diagram showing the fuzzy rule that determines theweighting coefficients for the input image and enhanced image.

FIG. 36 is a block diagram shows the overall construction of the imageprocessing apparatus of a tenth embodiment of the invention.

FIG. 37 is a block diagram showing the construction of the imagecombination unit of the tenth embodiment.

FIG. 38 is a flowchart showing the density correction process of theenhanced image of the tenth embodiment.

FIG. 39 is a concept diagram of the density correction process for theenhanced image of the tenth embodiment.

FIG. 40 is a flowchart explaining the image combination process of thetenth embodiment.

FIG. 41 is a block diagram showing the overall construction of the imageprocessing apparatus of an eleventh embodiment of the invention.

FIG. 42 is a block diagram showing the construction of the contrastimprovement unit of the eleventh embodiment.

FIG. 43 is a flowchart explaining the contrast improvement process ofthe eleventh embodiment.

FIG. 44 is a block diagram showing another example of the contrastimprovement unit of the eleventh embodiment.

FIG. 45 is a block diagram showing the construction of the contrastimprovement unit of a twelfth embodiment of the invention.

FIG. 46 is a flowchart explaining the contrast improvement process ofthe twelfth embodiment.

FIG. 47 is a diagram explaining the binding process.

FIG. 48 is a block diagram showing one example of the overallconstruction of the image processing apparatus of the twelfthembodiment.

FIG. 49 is a block diagram showing another example of the contrastimprovement unit of the twelfth embodiment.

FIG. 50 is a block diagram showing the construction of the imageprocessing apparatus of the thirteenth embodiment of the invention.

FIG. 51 is a schematic diagram showing an example of the pre-processing.

FIG. 52 is a diagram showing the construction of the contrastimprovement unit of the thirteenth embodiment.

FIG. 53 is a flowchart explaining the contrast improvement process ofthe thirteenth embodiment.

FIG. 54 is a block diagram showing the construction of the correctiondata calculation unit of the thirteenth embodiment.

FIG. 55 is a block diagram showing the construction of the imagecombination unit of the thirteenth embodiment.

FIG. 56 is a schematic diagram of the post-processing.

FIG. 57 is a block diagram showing the construction of the correctiondata calculation unit of a fourteenth embodiment of the invention.

FIG. 58 is a flowchart explaining the contrast improvement process ofthe fourteenth embodiment.

FIG. 59 is a block showing the construction of the correction datacalculation unit of a fifteenth embodiment of the invention.

FIG. 60 is a flowchart explaining the contrast improvement process ofthe fifteenth embodiment.

FIG. 61 is a block diagram showing the construction of the contrastimprovement unit of a sixteenth embodiment of the invention.

FIG. 62 is a schematic diagram for explaining the calculation of thecontrast improvement amount in the sixteenth embodiment.

FIG. 63 is a block diagram showing the construction of the contrastimprovement unit of a seventeenth embodiment of the invention.

FIG. 64 is a schematic diagram for explaining the calculation of thecontrast improvement amount in this seventeenth embodiment.

FIG. 65 is a flowchart explaining the contrast improvement process of aneighteenth embodiment of the invention.

FIGS. 66A and 66B are concept diagrams of the post-processing.

FIG. 67 is a diagram showing the construction of the post-processingunit in this eighteenth embodiment. And,

FIG. 68 is a flowchart explaining the post-processing in this eighteenthembodiment

BEST MODE FOR CARRYING OUT THE INVENTION First Embodiment

FIG. 4 is a block diagram showing an overview of the construction of theimage processing apparatus in the first embodiment of the invention.

In this image processing apparatus, after obtaining an analog imagesignal from the image pickup device, such as a CCD, the image input unit10 converts it to a digital image. The contrast improvement unit 11performs the contrast improvement process on the input image vi that wasinput from the image input unit 10. The image combination unit 12combines the enhanced image v obtained from the contrast improvementunit 11 and the input image vi from the image input unit 10. The imageoutput unit 13 outputs the combined image obtained from the imagecombination unit 12 to a desired device such as a printer or display asthe final output image (analog image signal) vo.

FIG. 5 is a block diagram showing an overview of the construction of thecontrast improvement unit.

In this contrast improvement unit 11, a correction data calculation unit100 calculates the amount of contrast improvement VRPij (Rr(i, j), Rg(i,j), Rb(i, j)) by comparing the three color components VPij (r(i, j),g(i, j), b(i, j)) at the object pixel Pij(i,j) with the pixelssurrounding the object pixel. The extraction unit 101 extracts aneffective range for that value from the distribution of the contrastimprovement amount VRPij obtained from the correction data calculationunit 100. The pixel value conversion unit 102 converts that contrastimprovement amount VRPij to the value on the enhanced image v accordingto the range extracted by the extraction unit 101.

FIG. 6 is a block diagram showing an overview of the construction of theimage combination unit.

In the image combination unit 12, the selection standard value judgmentunit 200 determines whether to give the input image vi or the enhancedimage v obtained from the contrast improvement unit 11 priority based onthe brightness of the input image vi. The combination coefficientcalculation unit 201 sets a combination coefficient w0 for the inputimage vi and a combination coefficient w1 for the enhanced image vobtained from the contrast improvement unit 11 based on the judgmentresults of the selection standard value judgment unit 200. The weightedaverage combination unit 202 uses the combination coefficients w0 and w1obtained from the combination coefficient calculation unit 122 togenerate a weighted average image of the input image vi and the enhancedimage v obtained from the contrast improvement unit 11.

This image processing apparatus will be explained in further detail. Inthis example, color image data is input to the contrast improvement unit11 from the image input unit 10.

Here, the image input unit 10 does not input the digitized components,red r, green g, and blue b, of the color image as they are to thecontrast improvement unit 11, but normalizes the data first beforeinputting it. When performing 8 bit digitization, the data of eachcomponent is given a value between 0 and 255, and the image input unit10 converts that value to a value between 0.0 and 1.0.

The contrast improvement unit 11 performs the contrast improvementprocess for that image data in order to improve the contrast in the darkareas of the input image. FIG. 7 is a concept diagram showing the modelof human visual perception that is used in this contrast improvementprocess.

As graphically shown in FIG. 7, humans do not visually perceive imagedata such as color or contrast by just the pixel value of the objectpixel Pij, but adjust the pixel value of the object pixel by therelative relationship between the data of the object pixel Pij and thedata of the surrounding pixels and visually perceive that image data.Humans, which have this kind of visual perception, can accuratelyrecognize the colors of an object even in scenes where there is unevenlighting such as when part has different lighting, or scenes where thereare large changes in intensity. In this embodiment, by using this kindof Retinex theory, colors and detail data in dark areas such as shadowsare clearly expressed.

FIG. 8 is a flowchart explaining the contrast improvement process.

When image data is input from the image input unit 10 to the contrastimprovement unit 11, the correction data calculation unit 111 calculatesthe maximum value Vmax (rx, gx, bx) and minimum value Vmin (rn, gn, bn)for each component from each of the pixel values in the input image vi(S1). When there is a large difference in these values for the inputimage vi and output image vo, there is a possibility that there will bea large sense of incongruity between both images. When trying tosuppress this sense of incongruity as much as possible, a process tomatch the minimum pixel values and maximum pixel values of both imagesshould be performed, and these values should be found in advance usingthis procedure.

The object pixel Pij is handled as the center view of human sight, andrectangular areas surrounding the object pixel Pij and having the sizeof c pixels is handled as the surround view. The correction datacalculation unit 100 finds the weighted average value VAPij (Ar(i, j),Ag(i, j), Ab(i, j)) for the pixel values in the area corresponding tothe surrounding area (S2). Furthermore, the correction data calculationunit 100 calculates the contrast improvement amount VRPij (Rr(i, j),Rg(i, j), Rb(i, j)) connected to the relative relationship between theweighted average value VAPij and the pixel value VPij (S3).

This weighted average value VAPij corresponds to the value of the seconditem in Equation 1. Similar to the conventional example, it is possibleto define that value by the convolution integral of the pixel value VPij(r(i, j), g(i, j), b(i, j)) corresponding to the surround view and thesurround view function F(x, y). However, in this embodiment, in order tosimplify and quicken the process, the value is defined by Equations 4below.

$\quad\begin{matrix}{{{{Ar}\left( {i,j} \right)} = {\sum\limits_{i}^{\;}\;{\sum\limits_{j}^{\;}\;{{r\left( {i,j} \right)}/C^{2}}}}}{{{Ag}\left( {i,j} \right)} = {\sum\limits_{i}^{\;}\;{\sum\limits_{j}^{\;}\;{{g\left( {i,j} \right)}/C^{2}}}}}{{{Ab}\left( {i,j} \right)} = {\sum\limits_{i}^{\;}\;{\sum\limits_{i}^{\;}\;{{b\left( {i,j} \right)}/C^{2}}}}}} & \left( {{Eqs}.\mspace{14mu} 4} \right)\end{matrix}$

From Equations 4, the value VAPij is given as the average of the pixelvalues of the rectangular areas corresponding to the surround view andhaving the size of c pixels.

At this time, the contrast improvement amount VRPij (Rr(i,j), Rg(i, j),Rb(i, j) is given by Equations 5 below.Rr(i,j)=r(i,j)/Ar(i,j)Rg(i,j)=g(i,j)/Ag(i,j)Rb(i,j)=b(i,j)/Ab(i,j)  (Eqs. 5)

The differences of the logarithmic transformation values are not used asin Equation 1. From Equations 5, the contrast improvement amount VRPijis given by the ratio of each of the components to the weighted averagevalue VAPij of the value VPij of the object pixel Pij.

The correction data calculation unit 100 finds the contrast improvementamount VRPij described above for all of the pixels of the input image vi(S4).

The extraction unit 101 finds the average value VaR (aRr, aRg, aRb) ofthe contrast improvement amount VRPij and the standard deviation valueVdR (dRr, dRg, dRb) for each component (S5). The extraction unit 101uses these values to calculate the minimum value emin and maximum valueemax from the distribution of the contrast improvement amount VRPij(S6).

Here, aRr+a*dRr, aRg+a*dRg, aRb+a*dRb are found as candidates for themaximum value emax, and the maximum value of these three values is takento be emax. Also, aRr−β*dRr, aRg−β*dRg, aRb−β*dRb are found ascandidates for the minimum value emin, and the minimum value of thesethree values is taken to be emin. It is also possible to find themaximum value emax and the minimum value emin by another method, howeverby using this method, it is possible to extract the necessary rangewithout disturbing the balance among the components of the contrastimprovement amount VRPij.

Next, the pixel value conversion unit 102 uses the maximum value emaxand the minimum value emin to convert the values of each components ofthe contrast improvement amount VRPij (Rr(i, j), Rg(i, j), Rb(i, j)) tovalues within the range 0.0 to 1.0 (S7). Furthermore, the pixel valueconversion unit 102 finds the pixel value on the enhanced image v fromthat conversion value. As described above, when the maximum value Vmax(rx, gx, bx) and the minimum value Vmin (rn, gn, bn) are found inadvance, they are used to keep the pixel value of the enhanced image vwithin the range of the maximum value Vmax (rx, gx, bx) and minimumvalue Vmin (rn, gn, bn) (S8).

After values, for which the contrast has been improved like this, areobtained for each object pixel Pij, the contrast improvement processends according to the end judgment by the pixel value conversion unit(S9).

Through this series of processes, just the portion near the center isobtained from the distribution of the ratio between the value of theobject pixel and the weighted average value of the pixels in the areacorresponding to the surround view, and the amount of change from thecenter is enhanced. Moreover, the contrast improvement amount VRPij ofpixels having the value of the ratio separated far from the area nearthe center becomes 1.0 or 0.0. Therefore, even in area where there is alittle difference between the object pixel corresponding to the centerview and the pixels in the area corresponding to the surround view, itbecomes easy to enhance that difference. In other words, the contrast isenhanced, and it becomes possible to clearly express details in shadowsor color data that is buried due to the insufficient range of the inputapparatus.

As explained above, calculation of the amount of contrast improvementfor each pixel is performed more easily than the method using theconventional Retinex theory. Moreover, in the method of the conventionalRetinex theory, there was a problem in that knowledge from experiencewas needed in order to set the filter process (offset, gain conversionfunction) when converting from the contrast improvement amount for eachpixel to the actual value on the image, however, in this embodiment itis not necessary.

However, when extracting just the portion near the center in thedistribution of the ratio between the value of the object pixel and theweighted average value of the values of the pixels in the areacorresponding to the surround view, and saturating the values in the twoportions straddling the center portion to 0.0 or 1.0, there is apossibility that there will be a problem in that the brightness level ofthe output image vo will all be concentrated near the center. Therefore,in this embodiment, in order to solve this problem and also further theproblem explained above in the conventional technology of a decrease inthe brightness level in very large highlighted areas having a uniformcolor, the input image vi and the enhanced image v are adaptivelycombined. By doing this, it is possible to suppress a decrease orincrease in the original brightness level of the input image vi.

After receiving the enhanced image v obtained from the contrastimprovement process and the input image vi, the image combination unit12 performs the process of combining both images and generating theoutput image vo according to the procedure shown in FIG. 9.

In the image combination unit 12, the selection standard value judgmentunit 200 calculates the brightness y(i, j) at the object pixel Pij inthe input image (S11). Also, it compares the brightness y(i, j) at theobject pixel Pij with the preset threshold values Th_h and Th_(—)1(S12).

In the case when Th_(—)1>y(i, j) or y(i, j)>Th_h, the selection standardjudgment unit 200 further finds the average brightness Ay(i, j) of thepixels in the area corresponding to the surround view of the objectpixel Pij (S13).

The selection standard value judgment unit 200 determines whether thebrightness y(i, j) satisfies the condition |Ay(i, j)−y(i, j)|<Th_len ornot (S14).

When the brightness y(i, j) does not satisfy the condition in step S14,or when the comparison result in step S12 shows that the brightness y(i,j) is between the thresholds Th_(—)1 and Th_h, the combinationcoefficient calculation unit 201 calculates the difference betweenbrightness y(i, j) at the object pixel Pij in the input image vi and thebrightness yd(i, j) at the pixel corresponding to the enhanced image v(S15). By substituting the absolute value of the difference between thetwo, len=|y(i, j)−yd(i, j)|, into the prepared coefficient settingfunction, the combination coefficient w1 for the enhanced image v is set(S16). Here, for the purpose of simplification, the threshold valuefunctions g(X) as shown in Equations 6 will be used for the settingfunction.g(x)=0.0(x>0.5)g(x)=0.5(x≦0.5)  (Eqs. 6)

The combination coefficient w0 for the input image is set by w0=1.0−w1.

After the combination coefficients w0 and w1 are set, the weightedaverage combination unit 202 uses these coefficients to find theweighted average value VWPij (WRr(i, j), WRg(i, j), WTb(i, j)) of thepixel value VPij in the input image vi and the pixel value VRPij of theenhanced image v (S16) for the object pixel Pij. The weighted averagecombination unit 123 sets the value found as a value at thecorresponding pixel in the output image vo (S17).

Also, when the brightness y(i, j) satisfies the condition in step S14, aprocess is performed to replace the value VWPij (WRr(i, j), WRg(i, j),WRb(i, j)) at the pixel Pij in the output image vo with the value VPij(r(i, j), g(i, j), b(i, j)) at the object pixel Pij in the input imagevi (S18). This process is a process that takes into consideration thatwhen the change in an area that is larger than the rectangular areacorresponding to the surround view and having size c is very small, itis easy for the contrast improvement amount for the pixel Pij toconcentrate near 1.0. This process removes large uniform highlightedareas (white) or black areas in the input image as being outside theprocess object. As a result, it is possible to suppress large changes inthe brightness level that can occur easily in areas such as mentionedabove in the contrast improved image.

By repeating this series of procedures for all pixels (S19), an outputimage vo with improved contrast is generated easily and high precisely.

The image processing described above, can similarly be realized by asoftware process using the central processing unit (CPU) and digitalsignal processor (DSP) used in a computer or the like, by following theimage processing method of this embodiment.

The contrast improvement amount VRPij (Rr(i, j), Rg(i, j), Rb(i, j)) canalso be given by Equations 7 below instead of Equations 5.Rr(i,j)=r(i,j)+γ×[r(i,j)/Ar(i,j)−1.0]Rg(i,j)=g(i,j)+γ×[g(i,j)/Ag(i,j)−1.0]Rb(i,j)=b(i,j)+γ×[b(i,j)/Ab(i,j)−1.0]  (Eqs. 7)

In this case, the contrast improvement amount VRPij (Rr(i, j), Rg(i, j),Rb(i, j)) is found by adding the value, obtained by subtracting 1.0 fromthe ratio of the value VPij of the object pixel Pij and the weightedaverage value VAPij and multiplying it by a preset positive constant γ,to the value VPij (r(i, j), g(i, j), b(i, j)).

By doing this, although the dependence on the set constant γ occurs,when the change in the value of the pixels in the area corresponding tothe surround view is very small, it is possible to prevent each of thecomponents of the contrast improvement amount VRPij (Rr(i, j), Rg(i, j),Rb(i, j)) from becoming a value near 1.0 regardless of the value VPij(r(i, j), g(i, j), b(i, j)).

Second Embodiment

The image processing apparatus of the second embodiment, hasconstruction that is similar to that of the image processing apparatusof the first embodiment, however, as shown in FIG. 10, the contrastimprovement unit 11 further comprises: an initial setting unit 103, anend judgment unit 104 and comparison range change unit 105.

The initial setting unit 103 performs the process of setting the size ofthe rectangular areas corresponding to the surround view. The endjudgment unit 104 determines whether or not a contrast improvementamount has been calculated for all of the preset rectangular areascorresponding to the surround view. The comparison range change unit 105changes the size c of the current area to the next candidate when it wasdetermined by the end judgment unit 104 that the process has not ended.

As shown graphically in FIG. 11, by performing contrast improvement onthe object pixel Pij corresponding to the center view according to theresult obtained by changing size c of the area corresponding to thesurround view, c0, c1, . . . ck, the bad influence caused by the size ofthe dark areas (shadows) in the image is decreased.

As shown in FIG. 12, after step S1 is performed, the initial settingunit 103 of the contrast improvement unit 11 sets the size c of the areacorresponding to the surround view, by selecting a size from a pluralityof preset sizes ck (k=0, 1, . . . , Cnum−1). The size of the areacorresponding to the surround view can be selected in ascending orderfrom the smallest size, or can be selected in descending order from thelargest size, however, it is preferred that the direction of size changebe the same. Here, the size is selected in descending order from thelargest size, so the area corresponding to the surround view issequentially decreased.

The correction data calculation unit 100 calculates the weighted averagevalue VAPij_k(Ar_k(i, j), Ag_k(i, j), Ab_k(i, j)) of the values of thepixels in the current area ck (S22).

The end judgment unit 104 determines whether calculation of the weightedaverage value has been calculated for all sizes (S23).

When it is determined that calculation has not been completed, thecomparison range change unit 105 changes the currently set area to thenext candidate. After changing the size of the area, the correction datacalculation unit 100 calculates the weighted average value for the areawith that changed size.

After calculation of the weighted average values is repeated like thisand the end is determined, the correction data calculation unit 100finds the weighted average value of the weighted average valuesVAPij_k(Ar_k(i, j), Ag_k(i, j), Ab_k(i, j)) for each size of area ck.Also, that value is set as the overall weighted average valueVAPij(Ar(i, j), Ag(i, j), Ab(i, j)) of the object pixel Pij (S24). Whendoing this, it is also feasible to perform weighting according to thesize ck of the area corresponding to the surround view, however, here,in order for simplification, the average value of the weighted averagevalues VAPij_k for each size ck is used as the overall weighted averagevalue for the object pixel Pij.

After the overall weighted average value VAPij has been found, thecorrection data calculation unit 100 calculates the ratio between thevalue VPij of the object pixel Pij and the overall weighted averagevalue VAPij as the contrast improvement amount VRPij (Rr(i, j), Rg(i,j), Rb(i, j)) (S25).

The process after a contrast improvement amount VRPij (Rr(i, j), Rg(i,j), Rb(i, j)) has been obtained for every pixel by repeating this kindof procedure is the same as in the first embodiment.

By performing this kind of process, while at the same time utilizing thefeatures of the image process of the first embodiment, it is possible toautomatically improve the contrast of the input image without beinggreatly affected by the dynamic range or the size of dark areas such asshadows, and thus as a result the contrast improvement becomes forefficient.

The image processing of this second embodiment also can be realized bysoftware process that uses a CPU and DSP.

Third Embodiment

In the image apparatus of the third embodiment, part of the contrastimprovement unit 11 has different construction than in the imageprocessing apparatus of the first embodiment. As shown in FIG. 13,instead of the correction data calculation unit 100, the contrastimprovement unit 11 comprises a signal conversion unit 106 and an objectcorrection data calculation unit 107. Also, instead of the pixel valueconversion unit 102, it comprises an object signal conversion unit 108and a signal inverse conversion unit 109.

Here, the signal conversion unit 106 converts the value VPij (r(i, j),g(i, j), b(i, j)) of a pixel Pij of a color image to another set ofsignals. The object correction data calculation unit 107 calculates theamount of contrast improvement in the same way as in the firstembodiment for one of those signals as the object. The signal inverseconversion unit 109 performs the inverse conversion process of theconversion process by the signal conversion unit 106 from signals otherthan the signal obtained by the object signal conversion unit 108 andthe object signal obtained by the signal conversion unit 106 to find thevalue of the pixel Pij on the enhanced image v.

The value VPij (r(i, j), g(i, j), b(i, j)) of a pixel Pij of a colorimage is converted to lightness L(i, j) and hue a*(i, j), b*(i, j), forexample. This lightness L(i, j) is the signal of the object for whichthe contrast is to be improved. The color data of the input image iskept as is while controlling the lightness level. This means that thecolor balance of the input image vi is reproduced as is in the enhancedimage v obtained by the contrast improvement unit 11.

The contrast improvement unit 11 performs the contrast improvementprocess by the procedure shown in more detail in FIG. 14.

After the contrast improvement process begins, the signal conversionunit 106 converts the value VPij (r(i, j), g(i, j), b(i, j)) of thepixel Pij in the input image vi to the value VPLij (L(i, j), a*(i, j),b*(i, j)) on the space La*b* (S31).

The object correction data calculation unit 107, extraction unit 101 andobject signal conversion unit 108 perform processing of the lightnessL(i, j) of the conversion value signals for the pixel Pij. During thatprocessing, the hue a*(i, j) and b*(i, j) are temporarily kept constant.Also, in order to accurately maintain the balance of the color data ofthe input image vi in the enhanced image v as well, it is preferred thatthe lightness L(i, j) be normalized to a value in the range 0.0 to 1.0.

The object correction data calculation unit 107 calculates the maximumvalue Lx and minimum value Ln of the lightness L(i, j) of the pixel Pijin the input image (S32). Furthermore, it finds the weighted averagevalue A(i, j) of the lightness (i, j) at the pixels in the areacorresponding to the surround view for the object pixel Pij (S33).Moreover, it calculates the ratio of the lightness L(i, j) and theweighted average value AL(i, j) as the contrast improvement amount RL(i,j) for the lightness (S34).

After the calculation of the contrast improvement amount RL(i, j) forthe lightness of the object pixel Pij has been repeated for all of thepixels in the input image vi, the extraction unit 101 finds the averagevalue aRL and standard deviation dRL of the contrast improvement amountRL(i, j) for the lightness (S36). Using those values, the extractionunit 101 sets the minimum value emin and maximum value emax from thedistribution of the contrast improvement amount RL(i, j) for thelightness (S37).

The object signal conversion unit 108 uses the minimum value emin andmaximum value emax to convert the contrast improvement amount RL(i, j)for the lightness to a value in the range from 0.0 to 1.0 (S38).Furthermore, the object signal conversion unit 108 finds the lightnessof the pixel in the enhanced image v from that conversion value (S39).This lightness is kept within the range of the maximum value Lx andminimum value Ln of the input image v.

The signal inverse conversion unit 109 uses the contrast improvementamount RL(i, j) for the lightness that was obtained in this way and thestored hue a*(i, j) and b*(i, j) to find the value VRPij (Rr(i, j),Rg(i, j), Rb(i, j)) of the pixel Pij of the enhanced image by usinginverse conversion of the signal conversion performed by the signalconversion unit 106 (S40). Also, in order to generate the enhanced imagev, this calculation is repeated for all of the pixels (S40 a).

By applying this series of processes, it becomes easier to maintain thecolor balance of the pixels in the input image than when independentlyperforming contrast improvement of the three components of the colorimage, and it is possible to keep the color in dark areas such asshadows in the improved image more stable. Furthermore, since thecontrast improvement amount is calculated for just the lightnesscomponent instead of performing it independently for the threecomponents, processing time can be greatly shortened.

In this embodiment, the signal conversion unit 106 converted the valueVPij of the pixel Pij of the input image vi to the value VPLij of theLa*b* space, however, the embodiment is not limited to this. It is alsopossible to convert the value VPij of the pixel Pij of the input imagevi to the luminance y(i, j), and color differences u(i, j) and v(i, j).Of these, by using the luminance y(i, j) as the object signal, it ispossible to perform the same image processing.

Also, the image processing in this third embodiment can be realized bysoftware processing using a CPU and DSP.

Fourth Embodiment

In the image apparatus of the fourth embodiment, part of the contrastimprovement unit 11 has different construction than in the imageprocessing apparatus of the first embodiment. As shown in FIG. 15, theconstruction of that contrast improvement unit 11 is a combination ofthe construction of the contrast improvement unit 11 of the secondembodiment and the construction of the contrast improvement unit 11 ofthe third embodiment. This means that the contrast improvement processexplained in the second embodiment is performed only for the objectsignal of the third embodiment, as shown in FIG. 16.

As in the third embodiment, the signal conversion unit 106 converts thevalue VPij (r(i, j), g(i, j), b(i, j)) of the pixel Pij in the inputimage vi to the value VPLij (L(i, j), a*(i, j), b*(i, j)) in the spaceLa*b* (S31). Of these converted signals, the hue a*(i, j) and b*(i, j)are maintained temporarily for later processing.

The initial setting unit 103 calculates the maximum value Lx and minimumvalue Ln of the lightness L(i, j) of the pixel Pij in the input image(S42). Furthermore, the initial setting unit 103 sets the size c of thearea corresponding to the surround view to a size selected from among aplurality of sizes ck (k=0, 1, . . . , Cnum−1).

The object correction data calculation unit 107 finds the weightedaverage value AL_k(i, j) of the lightness at the pixels in the currentarea ck (S43).

The end judgment unit 104 determines whether or not calculation of theweighted average value of the lightness has been completed for allsizes. When it is determined that calculation has not been completed,the comparison range change unit 105 changes the currently set area tothe next candidate. After the size of the area has been changed, theobject correction data calculation unit 107 calculated the weightedaverage value of the lightness for the area whose size has been changed.

After the calculation of the weighted average value of the lightness hasbeen repeated like this and end judgment is performed, the objectcorrection data calculation unit 107 finds the weighted average of theweighted average values for the areas ck of each size. It then sets thatvalue as the overall weighted average lightness AL(i, j). The objectcorrection data calculation unit 107 calculates the ratio of thelightness L(i, j) at the object pixel Pij and the overall weightedaverage lightness AL(i, j) as the contrast improvement amount RL(i, j)(S46).

The processing after obtaining a contrast improvement amount VRPij(Rr(i, j), Rg(i, j), Rb(i, j)) for every pixel by repeating thisprocedure is the same as that in the third embodiment.

By performing this kind of process, similar to the process in the secondembodiment, not only is there no need to preset an area corresponding tothe surround view, but as in the process in the third embodiment, itbecomes easy to maintain the color balance of the pixels, and it ispossible to better maintain the stability of color in dark areas such asshadows in the improved image. And, there is also the advantage that asthe process is simplified, it becomes possible to reduce the processingtime.

Of course, the image process in this fourth embodiment can also berealized with a software process that uses a CPU and DSP.

Fifth Embodiment

The image processing apparatus of the fifth embodiment of the inventionhas the same construction as the image processing apparatus in the firstembodiment, however, as shown in FIG. 17, the pixel value conversionunit 102 further comprises: an average brightness calculation unit 1020,a conversion method selection unit 1021 and pixel value estimation unit1022.

In that pixel value conversion unit 102, the average brightnesscalculation unit 1020 calculates the average brightness of the pixels inthe input image vi. Based on that average brightness value, theconversion method selection unit 1021 selects the method for convertingthe contrast improvement amount obtained by the extraction unit 101 tothe value of the enhanced image v. The pixel value estimation unit 1022uses the conversion method selected by the conversion method selectionunit 1021 and converts the contrast improvement amount obtained by theextraction unit 101 to the value of the enhanced image v.

In the contrast improvement unit 11, except for the pixel conversionunit 102, the correction data calculation unit 100 and extraction unit101 perform the same processing as in the first embodiment.

The correction data calculation unit 100 finds the contrast improvementamount for all of the pixels Pij in the input image vi. The extractionunit 101 extracts an effective range from the histogram distribution foreach component of the contrast improvement amount. After the range hasbeen extracted by the extraction unit 101 in this way, the pixel valueconversion unit 102 executes the pixel value conversion process as shownin FIG. 18.

The average brightness calculation unit 1020 calculates the brightnessy(i, j) of each pixel Pij in the input image vi, and finds the averagevalue ave_y (S61).

Based on this average value ave_y, the conversion method selection unit1021 selects the method for converting the contrast improvement amountto the actual pixel values on the enhanced image v. Many methods arepossible for this selection method, however, here a low brightnessthreshold Th low and high brightness threshold Th high are set, and aconversion function gg(x) is selected as shown in Equations 8 below (S62to S66).ave _(—) y<Th_low;gg(x)=k _(—) l·x ^(gl)(g _(—) l>1.0)ave _(—) y>Th_high;gg(x)=k _(—) h·x ^(gh)(g _(—) h<1.0)Th_low≦ave_(—) y≦Th_high;gg(x)=1.0/{1.0+exp(−g _(—) n·(x−0.5)}  (Eqs. 8)

In Equations 8, k_l, g_l, g_n, k_h and g_h are preset constants, and xis the value of the object component of the contrast improvement amountbefore conversion.

This conversion function sets the portion on the image to focus on basedon the average brightness value in the input image before processing.For example, when the overall brightness of the input image is low,instead of the entire image suddenly become bright, there is a betterimpression from a three-dimensional feel if the areas of low brightnessremain as much as possible. Also, when the overall brightness of theinput image is high, the human eye pays high attention to the brightareas, so it is necessary to maintain those areas as much as possible.Furthermore, when an image having an intermediate brightness, such as anature image, is output to a printer or the like, processing to makeareas that have a lower brightness than the center portion a littledarker and to make areas that have a higher brightness than the centerportion a little brighter is performed by gamma conversion. Theconversion method here follows this kind of insight, however the methodis not set to the method shown here, and any conversion method thatfollows this insight can be used in the same way.

After the conversion function has been selected, the pixel estimationunit 1022 uses that function to find the conversion value VRPoij (Rro(i,j), Rgo(i, j), Rbo(i, j)) for each component of the contrast improvementamount VRPij (Rr(i, j), Rg(i, j), Rb(i, j)) (S67, S68). Furthermore,from that conversion value, the pixel value estimation unit 1022 findsthe pixel value Pij on enhanced image v (S69). That value is keptbetween the maximum value Vmax (rx, gx, bx) and minimum value Vmin (rn,gn, bn) of each component of the pixel Pij in the input image vi. Also,the enhanced image v is obtained by repeating steps S68 and S69 for allof the pixels (S70).

By selecting the conversion method based on the average brightness ofthe input image as described above, it is possible to improve thequality of contrast improvement.

The image processing apparatus of this embodiment can also be combinedwith the image processing apparatus of the second embodiment.

Moreover, the image apparatus of this fifth embodiment can also berealized by a software process that uses a CPU and DSP.

Sixth Embodiment

The image processing apparatus of the sixth embodiment has the sameconstruction as the image processing apparatus of the third embodiment,however, as shown in FIG. 19, the object signal conversion unit 108further comprises: an average object signal calculation unit 1080, aconversion method selection unit 1081, and an object signal estimationunit 1082.

This kind of object signal conversion unit 108 selects the conversionmethod for an object signal instead of a pixel value as was done in thefifth embodiment, and then using the selected method, it finds theconversion value for the contrast improvement amount for the objectsignal obtained by the extraction unit 101.

After the contrast improvement amount for the object signal is input tothe object signal conversion unit 108 from the extraction unit 101, theaverage object signal calculation unit 1080 calculates the average valueof the object signals found from the value of the pixel in the inputimage vi. When the object signal is lightness, the average values forthe hue that corresponds to it are not calculated. In the fifthembodiment, the average value of the brightness was found, however inthis embodiment, only the average value of the lightness, which has thesame kind of meaning as the brightness, is calculated.

The conversion method selection unit 1081 selects the method forconverting the contrast improvement amount for the object signal thatwas obtained by the extraction unit 101 to the value of the objectsignal on the enhanced image vi based on the average value of thebrightness that was obtained by the average object signal calculationunit 1080. Except that the average value of the lightness is obtainedinstead of that of the brightness, the selection procedure is the sameas that in the fifth embodiment.

The object signal estimation unit 1082 finds the conversion value forthe contrast improvement amount for the object signal obtained by theextraction unit 101 according to the conversion method selected by theconversion method selection unit 1081.

After the object signal estimation unit 1082 finds the conversion value,the signal inverse conversion unit 109 generates the enhanced image fromthat conversion value and the stored hue values in the same way as inthe third embodiment.

By selecting the conversion method for the object signal in this way,not only is it possible to improve the stability of the color, but it isalso possible to perform more highly precise contrast improvement.

It is also possible for the object signal conversion unit 108 in thefourth embodiment to comprise: an average object signal calculation unit1080, a conversion method selection unit 1081, and an object signalestimation unit 1082.

Also, it is possible for the image processing apparatus of embodiment 6to be realized by a software process that uses a CPU and DSP.

Seventh Embodiment

In the image processing apparatus of the seventh embodiment of theinvention, the construction of part of the pixel value conversion unit102 differs from that in the image processing apparatus of the fifthembodiment. As shown in FIG. 20, that pixel value conversion unit 102comprises a standard intensity calculation unit 1023 and conversioncurve estimation unit 1024 instead of the average brightness calculationunit 1020 and conversion method selection unit 1021.

In that pixel value conversion unit 102, the standard intensitycalculation unit 1023 calculates the edge intensity of the brightnessthat expresses the contrast data of the input image vi. The conversioncurve estimation unit 1024 uses a generic algorithm to estimate aconversion curve for converting the contrast improvement amount obtainedby the extraction unit 101 to the value in the enhanced image v.

After the conversion curve has been estimated, the pixel valueestimation unit 1022 finds the conversion value for the contrastimprovement amount obtained by the extraction unit 101 based on theconversion curve instead of the selected conversion method.

FIG. 21 is a block diagram showing an overview of the construction ofthe conversion curve estimation unit 1024.

In the conversion curve estimation unit 1024, an initial candidatesetting unit 240 prepares an initial population of chromosomesindicating a plurality of conversion curves. A pixel value conversioncandidate calculation unit 241 finds a pixel value on each candidate forthe enhanced image v based on the conversion curve shown by eachchromosome. An evaluation value calculation unit 242 calculates anevaluation value for the conversion curves corresponding to eachcandidate from the brightness at the pixel on each candidate. A fitnesscalculation unit 243 calculates the fitness for each chromosome based onthe evaluation value obtained by the evaluation value calculation unit242. A recombination operation unit 244 performs recombination operationsuch as selection, crossover or mutation for the chromosomes containedin the chromosome population and generates the next generation of thechromosome population. An estimation end judgment unit 245 determineswhether or not optimum estimation has ended for the conversion curve,and as soon as estimation ends, that conversion curve is given to thepixel value estimation unit 1022.

In this embodiment, a genetic algorithm is used to give make the edgesections of the enhanced image v sharper. Also, since a suitableconversion function is automatically estimated, it is not necessary toset conditions beforehand.

FIG. 22 is a flowchart explaining the contrast improvement process ofthis embodiment.

As in the first embodiment, after the extraction unit 10 converts thevalues of each component of the contrast improvement amount VRPij (Rr(i,j), Rg(i, j), Rb(i, j)) to values within the range 0.0 to 1.0, the pixelvalue conversion unit 102 performs the pixel value conversion processS78, S79 including the conversion curve estimation process. FIG. 23 is aflowchart explaining in detail the pixel value conversion process.

The standard intensity calculation unit 1023 calculates the brightnessy(i, j) and color difference u(i, j), v(i, j) for each pixel Pij in theinput image vi (S81). Furthermore, from this brightness, it finds theedge intensity Ec of the input image vi (S82). This edge intensity isdefined by Equations 9 below.

$\quad\begin{matrix}{{{Ec} = {\sum\limits^{\;}\;{\sum\limits^{\;}\;{{ec}\left( {i,j} \right)}}}}{{{ec}\left( {i,j} \right)} = \left( {{{deli}\left( {i,j} \right)}^{2} + {{delj}\left( {i,j} \right)}^{2}} \right)^{1/2}}{{{delj}\left( {i,j} \right)} = {{- {y\left( {{i - 1},{j - 1}} \right)}} + {y\left( {{i - 1},{j + 1}} \right)} - {2.0*{y\left( {i,{j - 1}} \right)}} + {2.0*{y\left( {i,{j + 1}} \right)}} - {y\left( {{i + 1},{j - 1}} \right)} + {y\left( {{i + 1},{j + 1}} \right)}}}{{{deli}\left( {i,j} \right)} = {{- {y\left( {{i - 1},{j - 1}} \right)}} + {y\left( {{i + 1},{j - 1}} \right)} - {2.0*{y\left( {{i - 1},j} \right)}} + {2.0*{y\left( {{i + 1},j} \right)}} - {y\left( {{i - 1},{j + 1}} \right)} + {y\left( {{i + 1},{j + 1}} \right)}}}} & \left( {{Eqs}.\mspace{14mu} 9} \right)\end{matrix}$

In Equations 9, the edge intensity Ec is defined as the sum of the edgeintensities ec(i, j) in the pixels Pij calculated using a Sobel filter.

After the standard intensity calculation unit 1023 finds the edgeintensity Ec for the input image, the conversion curve estimation unit1024 performs the conversion curve estimation process that uses agenetic algorithm.

The genetic algorithm is described in detail in documents such as‘Genetic Algorithms in Search, Optimization and Machine Learning’ (DavidE. Goldberg, Addison Wesley). The genetic algorithm has superior globalsearch ability, and even though it may not be possible to obtaindifferentiation information from the evaluation function for the optimalsolution search problem, as long as it is possible to obtain theevaluation value itself, it is a useful method capable of performing asearch. It can also be applied to problems where the only judgmentstandard is human subjective evaluation.

In the conversion curve estimation unit 1024, the initial candidatesetting unit 241 prepares an initial population Gv, which has elements,Nch number of search vectors Vch[A] (S83). Here, the search vectorsVch[A] comprise n number of search parameters g0, g1, . . . , gn−1. Fromtheir relationship to living matter, the search vectors Vch[A] arecalled chromosomes, and the members of the search vectors Vch[A] (k=0, .. . , n−1) are called genes.

The chromosomes are made to correspond to the conversion curves forfinding conversion values in the range from 0.0 to 1.0 for the inputfrom the extraction unit 101. As shown in FIG. 24A, this conversioncurve can be expressed n number of coordinates (ik, ok) that correspondto the combination of the input value ik and output value ok. In thatcase, the genes gk, except for gene g0, are made to correspond to thedifference between the vector Ok from the origin to coordinate (ik, ok)and the vector Ok−1 from the origin to coordinate (ik−1, ok−1). The geneg0 is made to correspond to the vector O0 from the origin to thecoordinate (i0, o0). As shown in FIG. 24B, it is possible to performchromosome notation for the search vectors that correspond to theconversion curve by using a real coding scheme to arrange the n numberof genes g, g1, . . . , gk, . . . , gn−1 as are. Instead of the realcoding scheme, it is also possible to use a binary coding scheme inwhich the genes gk are notated as binary numbers and those binarynumbers are arranged in an array.

In order to prepare an initial population Gv of these kinds ofchromosomes, the initial candidate setting unit 240 selects n number ofreal numbers generated from uniform random numbers in the range from −1to 1, and sets one initial search vector v. By preparing the vectors ofthe same number as Nch number of preset chromosomes, the initialpopulation Gv={Vch[A]} (A=0, . . . , Nch−1) is set. The optimal searchstarts from this initial population Gv.

The pixel value conversion candidate calculation unit 241 uses theconversion curve corresponding to each of the chromosomes of set Gv tofind the conversion value VRPoij[A] (Rro(i, j, A), Rgo(i, j, A), Rbo(i,j, A)) from the input value VRPij (Rr(i, j), Rg(i, j), Rb(i, j))obtained by the extraction unit 101. The candidates for the enhancedimage v are found from this conversion value. Furthermore, the pixelvalue conversion candidate calculation unit 241 calculates thebrightness yo(i, j, A) for that conversion value.

The evaluation value calculation unit 242 finds the edge intensity E [A]for the candidates of the enhanced image v obtained from the conversioncurves according to that brightness yo as the evaluation value (S84).This evaluation value is the result of evaluating the appropriateness ofthe solution of the search vectors Vch[A] corresponding to theconversion curves according to a preset measure for evaluation. Thismeasure for evaluation is called the evaluation function (fitnessfunction). Here, Equations 9 gives the evaluation functions. This isbecause that an image with good visual contrast has the feature thatlight and dark in the image is not only very distinguishable but alsothe edges in image are sharp.

The fitness calculation unit 243 finds the maximum value yox[A] andminimum value yon[A] of the brightness yo(i, j, A) at each pixel in thecandidate of the enhanced image v obtained from the conversion curves(S85). Furthermore, from the evaluation value E[A] found by theevaluation value calculation unit 242, it calculates the fitness of thesearch vectors corresponding to the conversion curves (S86). Variousfunctions are possible as the functions for calculating the fitnessf[A]. Here the function given by Equation 10 is used. F is a constant.f[A]={E[A]+Φ*(yox[a]−yon[A])}/Ec  (Eq. 10)

The value obtained by adding, to the edge intensity E[A] of thatcandidate of the enhanced image v obtained from each conversion curve, avalue that is related to the distance between the maximum value yox[A]and the minimum value yon[A] of the candidate is divided by the edgeintensity Ec of the input image vi to give the fitness f[A]. The termrelated to the distance between the maximum value yox[A] and the minimumvalue yon[A] not only is used for improving the edge intensity, but isalso used to contain as many conversion output areas as possible.

The larger the value for the fitness f[A] is, the better the searchvector, or in other words, the conversion curve is determined to be.

The recombination operation unit 244 performs a selection for eachchromosome of the population Gv according to this fitness f[A] (S87),and in order to generate a next generation population from that set Gv,it performs genetic recombination operation of the selected chromosomessuch as cross over (S88) and mutation (S89). Here, the number Nch ofchromosomes of each generation is fixed, however the number can beincreased or decreased.

The selection is performed by the Roulette selection method of selectinga search vector with probability that is proportional to the fitness. Asummary of processing by the Roulette selection method is given in (i)to (iii) below.

(i) The sum F of the fitness of all of the search vectors is found usingEquation 11 below from the fitness f[A] of the search vectors Vch[A](A=0, . . . , Nch−1) of the population Gv.

$\begin{matrix}{F = {\sum\limits_{j = 1}^{n}\;{{f\lbrack A\rbrack}\mspace{14mu}\left( {{k = 1},\ldots\mspace{14mu},{Nch}} \right)}}} & \left( {{Eq}.\mspace{14mu} 11} \right)\end{matrix}$

(ii) The selection probability h[k] for selecting a search vector Vch[A]as a parent for creating the next generation search vectors is setaccording to Equation 12 below.h[A]=f[A]/F  (Eq. 12)

(iii) A selection range I[A] for each search vector is assigned in thespace [0, 1] according to Equations 13 below, for example.

$\begin{matrix}\begin{matrix}{{{Ih}\lbrack 0\rbrack} = 0} \\{{{Ih}\lbrack A\rbrack} = {\sum\limits_{j = 0}^{A}\;{h\lbrack j\rbrack}}} \\{{I\left\lbrack {A - 1} \right\rbrack} = {\left\lbrack {{{Ih}\left\lbrack {A - 1} \right\rbrack},{{Ih}\lbrack A\rbrack}} \right\rbrack\mspace{31mu}\left( {{A = 1},\ldots,\;{Nch}} \right)}}\end{matrix} & \left( {{Eq}.\mspace{14mu} 13} \right)\end{matrix}$

Also, a set RR of uniform random numbers ra[A] is generated with [0, 1]where RR=(ra[0], ra[1], . . . , ra[Nch−1]). Here 0<ra[j]<1.

Also, a set Num=(num[0], num[1], . . . , num[Nch−1]) in is found. Eachelement of Num, num[B] (B=0, 1, . . . , Nch−1), is defined as theselection range I[A] which satisfies Equation 14 below. In this way theset of Nch number of search vectors corresponding to the set Num isselected.ra[BA]εI[A](A,B=0,1, . . . , Nch−1)  (Eq. 14)

Using this kind of Roulette selection method, new next generation searchvectors Vch[A] are selected from the current population of searchvectors Gv.

After the chromosomes have been selected as the parent of new nextgeneration chromosomes, cross over processing and mutation processingare performed on them as shown in FIGS. 25A and 25B. Cross over is anoperation of creating a new search vector by exchanging parts of twochromosomes expressed by finite symbols. This operation corresponds tosearching in a location separated far from the current solution vector.In the example in FIG. 25A, the symbols between cross over point 1 andcross over point 2 of chromosome Vch[A] and chromosome Vch[B] areexchanged. The cross over here can also be one-point cross over ortwo-point cross over.

Also, mutation is an operation of selecting at random a symbol in partof the chromosome having low probability, and changing to selectedsymbol with another symbol. This operation corresponds to searching nearthe current search vector. In the example shown in FIG. 25B, of thesymbols of the chromosome, the symbol g4 [A] is mutated to f4 [A]. Bychanging the probability of performing mutation using half of thepopulation and the remaining half, it is possible to better maintain thediversification of the search vector.

The population of next generation search vectors is created by this kindof cross over and mutation processing.

When the estimation end judgment unit 245 determines that theconvergence condition has not yet been satisfied, the steps S84 to S89are repeated. That convergence condition can be a condition such as whenthe number of repetitions exceeds the allowable number of repetitions.

By repeating this process, estimation of search parameters of theoptimum conversion curve is performed. Many candidates are globallyprepared, and by repeatedly performing a symbol recombination operationsuch as cross over or mutation that are not trapped easily by the localoptimum solution (local minimum), it becomes possible to automaticallyand accurately estimate the optimum conversion curve.

After the optimum conversion curve is obtained, the pixel valueestimation unit 245 uses that conversion curve to find the conversionvalue in the range of 0.0 to 1.0, and finds the value of the pixel onthe enhanced image v (S79). The value of the pixel in the enhanced imagev is kept within the maximum value Vmax (rx, gx, bx) and the minimumvalue Vmin (rn, gn, bn) of each of the components of the pixel Pij inthe input image vi.

As described above, in the image processing apparatus of thisembodiment, by using as an evaluation value improvement degree of theedge intensity related to the articulation of detailed areas whenestimating the conversion function, it is possible to improve thecontrast and also improve the sharpness of the edges. Also, by using agenetic algorithm in the estimation of the optimum conversion function,it is possible to automatically estimate the optimum conversion functionwithout depending on the input image.

It is also possible to combine the image processing of this embodimentwith the image processing of the second embodiment.

Moreover, the image processing of this seventh embodiment can berealized by a software process that uses a CPU and DSP.

Eighth Embodiment

In the image processing apparatus of the eighth embodiment, part of theobject signal conversion unit 108 has different construction than thatof the image processing apparatus of the sixth embodiment. As shown inFIG. 26, that object signal conversion unit 108 comprises a standardintensity calculation unit 1083 and object signal conversion curveestimation unit 1084 instead of the average object signal calculationunit 1080 and conversion method selection method 1081.

When the contrast improvement amount for the object signal is input fromthe extraction unit 101 to that object signal conversion unit 108, thestandard intensity calculation unit 1083 calculates the edge intensityfor the object signal that expresses the contrast data of the inputimage vi. The object signal is the lightness, for example.

The object signal conversion curve estimation unit 1084 uses a geneticalgorithm to estimate the object signal conversion curve for convertingthe contrast improvement amount for the lightness to the value in theenhanced image v. As shown in FIG. 27, except for having an objectsignal conversion candidate calculation unit 246 instead of the pixelvalue conversion candidate calculation unit 241, the object signalconversion curve estimation unit 1084 has the same construction as theconversion curve estimation unit 1024.

In the object signal conversion curve estimation unit 1084, after theinitial candidate setting unit 246 prepares an initial population, theobject signal conversion candidate calculation unit 246 finds the valuefor the lightness of the candidate for the enhanced image v based on theobject signal conversion curve indicated by the prepared population ofchromosomes. The evaluation calculation unit 242 calculates anevaluation value for the object signal conversion curves correspondingto each candidate from the found values for the lightness of eachcandidate. The fitness calculation unit 243, recombination operationunit 244 and estimation end judgment unit 245 perform the sameprocessing as in the seventh embodiment except that when the estimationend judgment unit 245 determines the end of estimation, the objectsignal conversion curve is given to the object signal estimation unit1082.

After the object signal conversion curve has been estimated, the objectsignal estimation unit 1082 finds the conversion value for the contrastimprovement amount for the object signal obtained by the extraction unit101 based on the object signal conversion curve instead of the selectedconversion method.

As described above, in the image processing of this embodiment, byperforming the same process as in the seventh embodiment for the objectsignal, which in this case is the lightness, not only is it possible toimprove the contrast and the sharpness of the edges, but it is alsopossible to improve the color stability of pixels in dark areas such asshadows.

It is also possible to combine the image processing of this embodimentwith the image processing of the fourth embodiment.

Moreover, the image processing of this eighth embodiment can also berealized by a software process that uses a CPU and DSP.

Ninth Embodiment

On problem with images taken with a digital camera is the phenomenon ofcolor cast. This is a phenomenon where due to the photographicconditions, certain colors come out stronger than what is proper. Humansare capable of adapting to light, so seeing different colors due todifferences in the light source does not often occur, however, in animage taken with a digital camera, color cast occurs a little due to thelight source when the image was taken. At sunrise and sunset there isred color cast, and under fluorescent light, there is a tendency for thecolor to turn a green color. Also, the color cast occurs by reflectionfrom the circumference of the subject of a photograph, for examplereflection by water in a pool causes the blue color cast to occur, andreflection by plants causes the green color cast to occur.

In order to remove this kind of color cast, for example, when the colorgreen G is to strong, correction of the tone curve and density level inorder to weaken the cyan C and yellow Y signals is performed manually.

Also, there is auto white balance processing that performs adjustmentautomatically based on the Evans theory. The Evans theory is a theorybased on the assumption that when all of the photographed objects in theworld are added the result is achromatic color, the basis of thisprocessing is to adjust the balance of all colors such that the averagevalue of the entire input image is gray (achromatic color).

FIG. 28 is a block diagram showing one example of the construction of anapparatus for performing this kind of process automatically. In thisapparatus, the input image distribution detection unit 2301 finds theaverage value for the red, blue and green signals in the input imagefrom the image input apparatus 2301. The standard color setting unit2303 calculates the average value of the brightness of the input imageand sets this value as the standard color. The color balance correctionunit 2304 corrects each of the signals such that the average value ofeach of the red, blue and green signals obtained by the input imagedistribution detection unit 2303 becomes the standard color. With thiscorrection, the average color of the entire input image becomes gray andcolor cast is automatically reduced.

However, when this kind of process is used for the input image and thereis an object in that input image that has a very vivid color thatoccupies a large area, the tendency for that color to become the averagecolor becomes strong, and there is a problem that this color will becomeclose to the achromatic color. For example, when the color green ofgrass or lawn occupies a large part of the image, a phenomenon (colorfailure) occurs in which the color of the grass becomes the achromaticcolor.

In this ninth embodiment, image processing is explained in which thecontrast is improved and the input image and enhanced image areadaptively combined such that this kind of color cast is reduced.

As shown in FIG. 29, in addition to an image input unit 10, contrastimprovement unit 11, image combination unit 12 and image output unit 13,the image processing apparatus of this ninth embodiment comprises anedge data detection unit 14 that detects edge data from the input imagevi.

Also, as shown in FIG. 30, in addition to the combination coefficientcalculation unit 201 and weighted average combination unit 202, theimage combination unit 12 comprises an output value setting unit 203,but does not comprise a selection standard value judgment unit 200.Here, the combination coefficient calculation unit 200 sets combinationcoefficients w0 and w1 based on the edge data obtained by the edge datadetection unit 14 and not on the judgment results of the selectionstandard value judgment unit 201. The output value setting unit 203compares the combined image obtained by the weighted average combinationunit 202 and the input image vi to set the value of the pixels in theoutput image vo.

FIG. 31 is a flowchart explaining the process of detecting edge data.

After image data is input from the image input unit 10, the edge datadetection unit 14 initializes the position of the object pixel and thencalculates the brightness y(i, j), for example, of the pixel Pij in theinput image vi. This calculation is repeated for all of the pixels toobtain the brightness y(i, j) for all of the pixels Pij in the inputimage vi. Furthermore, the edge data detection unit 14 initializes theposition of the object pixel again, and then calculates the edge databased on the calculated brightness. It is possible to detect edge datafor each value of the three color components, however, by detecting theedge data for Just the brightness, it is possible to simplify theprocess. There are various methods for detecting edge data, however,here 5 pixel×5 pixel sized filters, such as shown in FIG. 32, will beused. Pixel Pij is taken to be the center, and each filter is appliedand the square root of the sum of the squares of the component amountex(i, j) obtained in the x direction and the component amount ey(i, j)obtained in the y direction is taken to be the edge data ed(i, j) ofthat pixel Pij. The filter size and type are not limited to those ofthis example, and it is also possible to use the previously described 3pixel×3 pixel sized Sobel filter.

The edge data detection unit 14 repeats the process described above fordetecting edge data for all of the pixels in the input image vi, andfrom that obtains the edge data for each pixel Pij of the input imagevi. Furthermore, the edge data detection unit 14 finds the average valueave_y of the brightness and the average value ave_ed of the edge databased on the brightness and edge data of each pixel Pij.

Image data that is the same as that input to the edge data detectionunit 14 is also input to the contrast improvement unit 11. The contrastimprovement unit 11 generates an enhanced image the same as done in thefirst embodiment except that it uses Equations 15 and 16 below insteadof Equations 4 and 5.

$\quad\begin{matrix}{{{Ay}\left( {i,j} \right)} = {\sum\limits_{i}^{\;}\;{\sum\limits_{j}^{\;}\;{{y\left( {i,j} \right)}/C^{2}}}}} & {\left( {{Eq}.\mspace{14mu} 15} \right)\mspace{14mu}} \\{{{{Rr}\left( {i,j} \right)} = {{r\left( {i,j} \right)}/{{Ay}\left( {i,j} \right)}}}{{{Rg}\left( {i,j} \right)} = {{g\left( {i,j} \right)}/{{Ay}\left( {i,j} \right)}}}{{{Rb}\left( {i,j} \right)} = {{b\left( {i,j} \right)}/{{Ay}\left( {i,j} \right)}}}} & \left( {{Eqs}.\mspace{20mu} 16} \right)\end{matrix}$

Equation 15 defines the average value Ay(i, j) of the brightness y(i, j)of the pixels in the c pixel sized area corresponding to the surroundview. Also, Equations 16 define the contrast improvement amount VRPij(Rr(i, j), Rg(i, j), Rb(i, j)) by the ratio of each component of thepixel Pij and the average value Ay(i, j) instead of the values of eachcomponent of the weighted average value VAPij.

By defining the contrast improvement in this way, the comparison valuefor each component value is the same. Here, it is possible to define thecontrast improvement as in Equations 4 and 5, however, calculating thecontrast improvement amount using a common value makes it possible tobetter maintain balance of the obtained contrast improvement amountsthan when calculating the contrast improvement amounts for eachcomponent independently.

The contrast improvement unit 11 generates an enhanced image v, and whenthat enhanced image v is input to the image combination unit 12, thecombination coefficient calculation unit 201 initializes the position ofthe object pixel Pij, as shown in FIG. 33, and then sets combinationcoefficients w0 and w1 for both images. In this embodiment, the inputimage vi and enhanced image v are adaptively combined by controlling thecombination coefficients based on the brightness and edge data of thepixels in the input image vi. Setting the combination coefficients isperformed according to fuzzy Rules 1 and 2 below.

(Rule 1)

-   -   if y(i, j) is Low and ed(i, j) is Low or y(i, j) is High and        ed(i, j) is Low then VWP(i, j) is VP(i, j)

(Rule 2)

-   -   if y(i, j) is High and ed(i, j) is High or y(i, j) is Middle and        ed(i, j) is Middle then VWP(i, j) is VRP(i, j)

In the example of the input image vi shown in FIG. 34, Rule 1 is therule that corresponds to area A1 or A2. Rule 2 is the rule thatcorresponds to area A3 or A4. In an areas such as area A1 where thebrightness of the pixels is low and change is smooth, or in an area suchas A2 where the brightness of the pixels is high and the change issmooth, there is a possibility that color noise or compression noisewill appear due to a decreases in brightness and sudden increases inbrightness in the enhanced image. In other areas such as area A3 and A4this problem does not occur in the enhanced image, and color cast inthat enhanced image is also decreased.

Therefore, when the degree of reliability ml of Rule 1 is large, thevalue of the pixels in the input image vi are given more influence onthe values of the pixels of the combined image than the values of thepixels in the enhanced image v, to prevent a drop in brightness or toprevent color noise or compression noise from occurring. Moreover, whenthe degree of reliability m2 of Rule 2 is large, the value of the pixelsin the enhanced image v are given more influence on the values of thepixels of the combined image than the values of the pixels in the inputimage vi, to improve the contrast and decrease color cast.

The degrees of reliability m1 and m2 are given as shown in FIG. 35 forexample. The degree of reliability m1 can be defined by a peak shapedfunction such as a Gauss function having two peaks; one at the pointwhere the edge data is small and brightness is high and the other wherethe brightness is low. Also, the degree of reliability m2 can be definedby a peak shaped function where the average value ave_ed of the edgedata and the average value ave_y of the brightness of the input imageare at the center.

The combination coefficient calculation unit 201 sets the combinationcoefficient w1 for the enhanced image v based on the degree ofreliability m1 and m2 such that w1=m2/(m1+m2). Also, it sets thecombination coefficient w0 for the input image v1 such that w0=1.0−w1.

The weighted average combination unit 202 finds the value VWPij (Wr(i,j), Wg(i, j), Wb(i, j)) of the object pixel Pij in the combined imagebased on the set combination coefficients w0 and w1.

By controlling the combination coefficients such that priority is givento the input image vi in areas not near the edge, but where thebrightness is high or low, and priority is given to the enhanced image vin other areas near the edges, not only the contrast is improved butcolor cast is suppressed. Also, it is possible to prevent a decrease ofbrightness in highlighted areas and prevent color noise or compressionnoise from occurring in shadow areas, and it is also possible tosuppress the possibility of discontinuity that could occur in thecombined image.

However, even when the values of the pixels in the combined image areset in this way, there is a possibility that there will be a decrease inbrightness when compared with the pixel values of the input image.

Therefore, the output value setting unit 203 compares the value VPij(r(i, j), g(i, j), b(i, j)) of the object pixel Pij in the input imagevi with the value VWPij (Wr(i, j), Wg(i, j), Wb(i, j)) of the objectpixel of the combined image. As a result, when the value VWPij is lessthan the value VPij, the output value setting unit 203 replaces thevalue VWPij with the value VPij. When the value VWPij is not less thanthe value VPij, replacement is not performed.

It is also possible to set w0=1.0 and w1=0.0 when the combinationcoefficient calculation unit 201 compares the pixel value VPij of theinput image vi with the pixel value VRPij of the enhanced image v andall of the three components of the value VPij are not less than thevalue VRPij. In other cases it sets w1=w3=0.5.

Doing this simplifies the process of setting the combinationcoefficients. However, particularly in areas that are not edges, in acombined image where the coefficient wO is larger than in the exampleabove, the percentage of the image occupied by the input image becomeslarge. Therefore, the image receives the bad influence of color cast ofthe input image.

Also, the image processing of this ninth embodiment can be combined withthe image processing of the second embodiment.

In the second embodiment, the ratio of the value VPij of the objectpixel Pij with the overall weighted average value VAPij was used as thecontrast improvement amount, however, it is also possible to used theratio of the ratio of the value VPij with the maximum weighted averagevalue. The maximum weighted average value is found by comparing theweighted average values of each size ck for each component. The time forthe comparison process increases, however, there are advantages in thatthe fluctuation of the pixel values in the enhanced image v becomesgradual, and sudden increases in values near the edges are prevented.

Furthermore, the image processing of this ninth embodiment can also berealized by a software process that uses a CPU and DSP.

Tenth Embodiment

As shown in FIG. 36, in addition to an image input unit 10, contrastimprovement unit 11, image combination unit 12 and image output unit 13,the image processing apparatus of a tenth embodiment of the inventioncomprises a density correction unit 15 that corrects the densitydistribution of the enhanced image v according to the densitydistribution of the input image vi.

Moreover, the image combination unit 12, as shown in FIG. 37, comprisesa weighted average combination unit 202 and output value setting unit203 as in the ninth embodiment, however, it does not comprise acombination coefficient calculation unit 201. This simplified imagecombination unit 12 combines the corrected image obtained by the densitycorrection unit 15 and the input image.

FIG. 38 is a flowchart explaining the density correction process.

After the enhanced image v is obtained by the contrast improvement unit11, the density correction unit 15 initializes the position of theobject pixel Pij and then calculates the luminance Ry(i, j), and colordifferences Ru(i, j), Rv(i, j) of the object pixel Pij in the inputimage vi and enhanced image v.

After this calculation is repeated for all of the pixels in the inputimage vi and enhanced image v, the density correction unit 15 calculatesthe average luminance ave_y of the input image vi and the averageluminance ave_Ry of the enhance image v from these values.

After calculating the average luminance ave_y and ave_Ry, the densitycorrection unit 15 initializes the position of the object pixel Pij inthe enhanced image v, and performs correction by adding the differencebetween the average luminance values ave_y-ave_Ry to the luminance Ry(i,j) of the object pixel Pij in the enhanced image v.

After correcting the luminance Ry(i, j), the density correction unit 15calculates the corrected value VDPij (Dr(i, j), Dg(i, j), Db(i, j)) ofthe object pixel Pij in the enhanced image v from the correctedluminance Ry(i, j) and color differences Ru(i, j), Rv(i, j).

The corrected image is obtained by calculating a correction value foreach pixel in the enhanced image v. In areas where the change inluminance is small over a wide area, there is a possibility that theluminance will decrease or suddenly increase by the contrastimprovement. As a result, the average luminance in the enhanced imageave_Ry tends to be affected largely. Therefore, as shown in FIG. 39, thecorrection value is calculated such that the average luminance ave_Ry ofthe enhanced image v matches the average luminance ave_y of the inputimage while saving the luminance distribution of the image with enhanceddetail. By doing this, decreases in luminance in highlighted areas andsudden increases in luminance in shadows are suppressed in the correctedimage.

As shown in FIG. 40, when the corrected image is input to the imagecombination unit 12 from the density correction unit 15, the weightedaverage combination unit 202 initializes the position of the objectpixel Pij, then uses the combination coefficient w0 and w1 to calculatethe value VWPij of the combined image from the value VPij of the inputimage and the value VRPij of the corrected image.

As already described, decreases in luminance in large highlighted areasare suppressed in the corrected image, so there is little need to useedge data to calculate the combination coefficients w0 and w1. It isalso possible to use preset values for the combination coefficients w0and w1 such that the values of the corrected image have priority of theinput image. For example, the combination coefficient w0 for the inputimage is set in advance to 0.3, and the combination coefficient w1 forthe corrected image is set to 0.7.

Moreover, the output value setting unit 203 performs the same procedureas in the ninth embodiment. The combined image is obtained from this.

By using the corrected image instead of the enhanced image as describedabove, it is possible to more easily obtain a combined image withimproved contrast while at the same time suppressing decreases inluminance in highlighted areas and sudden increases in luminance inshadows. Furthermore, since the percentage of the output image occupiedby the input image is reduced, it is possible to suppress the effect ofcolor cast in the input image from appearing in the output image.

The image processing of this tenth embodiment can also be realized by asoftware process that uses a CPU and DSP.

Eleventh Embodiment

As shown in FIG. 41, the image processing apparatus of an eleventhembodiment of the invention is similar to the image processing apparatusof the ninth embodiment in that it comprises an edge data detection unit14, however, the edge data found by that unit is used by the contrastimprovement unit 11 instead of the image combination unit 12.

Also, as shown in FIG. 42, the contrast improvement unit 11 comprises acorrection data calculation unit 100, extraction unit 101 and pixelvalue conversion unit 102 as in the image processing apparatus of thefirst embodiment, and further comprises an area judgment unit 110,comparison range selection unit 111, adjustment coefficient calculationunit 112 and adjustment unit 113.

When edge data ed(i, j) is input from the edge data detection unit 14,the contrast improvement unit 11 performs a process of generating theenhanced image v according to the procedure shown in FIG. 43.

In that contrast improvement unit 11, the area judgment unit 110initializes the position of the object pixel Pij, then determines thearea in which that pixel exists based on the edge data ed(i, j) andbrightness y(i, j) of the object pixel Pij. That area includes: shadowareas, such as area A1 in FIG. 34, where there is low brightness andsmooth change; highlighted areas, such as area A2, where there is highbrightness and smooth change; and areas near edges such as areas A3 andA4.

The comparison range selection unit 111 selects the size of the areacorresponding to the surround view according to the judgment result. Inthe case where it is determined that the pixels exists in a shadow areawith low brightness and smooth change or a highlighted area with highbrightness and smooth change, the comparison range selection unit 111selects a large size in order to suppress over enhancement or decreasesin brightness compared with the input image. In the case where it isdetermined that the pixel is in an area near an edge, it selects a smallsize in order to enhance changes that are buried in the input image. Inthe case where it is unclear where the pixels exist, it selects anintermediate size.

Also, the adjustment coefficient calculation unit 112 sets an adjustmentcoefficient according to the judgment result of the area judgment unit110 for adjusting the contrast improvement amount obtained by thecorrection data calculation unit 100. Of the areas described above, inareas with high brightness and smooth change, the contrast improvementamount VRPij tends to be near 1.0 and is expected to become lower thanthe value of the input image, so an adjustment coefficient k(i, j) isset in order to increase the contrast improvement amount. Here, theadjustment value k(i, j) is set to 1.2 in order to increase the contrastimprovement amount by 20%. For areas where the brightness is low andthere is smooth change, the contrast improvement amount VRPij tends tobe near 1.0 and is expected to be abruptly higher than the value of theinput image, so an adjustment coefficient k(i, j) is set in order todecrease the contrast improvement amount. Here, the adjustment valuek(i, j) is set to 0.8 in order to decrease the contrast improvementamount by 20%. For areas near edges, the adjustment coefficient k(i, j)will be set to 1.0.

After the adjustment coefficient k(i, j) is set by the adjustmentcoefficient calculation unit 112, and the contrast improvement amountVRPij is found by the correction data calculation unit 100, theadjustment unit 113 multiplies each component of the contrastimprovement amount VRPij by the corresponding adjustment coefficientk(i, j) to adjust the contrast improvement amount VRPij.

After repeating this series of processes and adjusting the contrastimprovement value VRPij for each pixel, the extraction unit 101 andpixel value conversion unit 102 perform the same procedure as in thefirst embodiment.

By adjusting the contrast improvement amount in this way, it is possibleto prevent the brightness of highlighted areas in the enhanced imagefrom becoming less than that of the input image and to prevent excessiveenhancement in shadow areas. Therefore, as in the tenth embodiment, itis possible to more easily combine images. Furthermore, since thepercentage of the output image that the input image occupies becomessmaller, it is possible to prevent the bad influence of color cast ofthe input image from appearing in the output image.

In the case of combining images in the same way as in the ninthembodiment, the image combination unit 12 as well as the contrastimprovement unit 11 can use the edge data found by the edge datadetection unit 14.

Also, as shown in FIG. 44, it is possible to combine the construction ofthe contrast improvement unit 11 of the image processing apparatus ofthis embodiment with the construction of the contrast improvement unit11 of the second embodiment.

In this case, the adjustment unit 113 adjusts the contrast improvementamount calculated using the weighted average APij k for a plurality ofsized areas ck.

By doing this, it is possible to reduce the effects of misjudged sizeselection. Also, when calculating the contrast improvement amount forjust values near the pixels in the edge area, there is a possibilitythat the color white will stick out due to excessive enhancement,however it is possible to decrease the amount that it sticks out nearthe edges. Furthermore, when the selected size is too small for the sizeof the shadow, the value of the object pixel will approach the value ofthe surrounding pixels and there is a possibility that enhancement willbe insufficient, however it is possible to reduce that possibility.

Also, the image processing of this eleventh embodiment can be realizedby a software process that uses a CPU and DSP.

Twelfth Embodiment

In the image processing apparatus of a twelfth embodiment of theinvention, the construction of part of the contrast improvement unit 11is different than that in the first embodiment. As shown in FIG. 45, inaddition to a correction data calculation unit 100, extraction unit 101and pixel value conversion unit 102, the contrast improvement unit 11comprises a correction data calculation density binding unit 114.

The correction data calculation standard binding unit 114 binds valuesof the pixels in the area corresponding to the surround view surroundingthe object pixel. Also, the correction data calculation unit 100 usesthe value of the bound pixels to find the contrast improvement amountfor the object pixel.

As shown in FIG. 46, the procedure of this contrast improvement processis nearly the same as that of the ninth embodiment, however, it differsin that the contrast improvement amount is found with the upper andlower limits of the values of the pixels in the area corresponding tothe surround view of the object pixel Pij being controlled.

Here, the case of calculating the contrast improvement amount VRPij(Rr(i, j), Rg(i, j), Rb(i, j)) using Equations 15 and 16 will beconsidered. In that case, as shown in FIG. 47, the correction datacalculation density binding unit 114 applies a lower limit thLow_y andupper limit thigh_y to the brightness y(i, j) of the pixels in the areacorresponding to the surround view of the object pixel Pij. Whencalculating the contrast improvement amount, values that exceed theupper limit are handled as being the same as the upper limit, and valuesless than the lower limit are handled as being the same as the lowerlimit.

When finding the contrast improvement amount for pixels in uniformlylarge highlighted areas by controlling the upper limit of the brightnessy(i, j) in this way, the average value Ay(i, j) is kept less than thebrightness y(i, j) of the object pixel Pij. As a result, the contrastimprovement amount VRPij becomes a value larger than 1.0, which is thevalue in the center. Therefore, the values of the enhanced imageobtained by the pixel value conversion unit 102 become high.

On the other hand, when finding the contrast improvement amount forpixels in uniformly large shadow areas by controlling the lower limit ofthe brightness y(i, j), the average value Ay(i, j) increases more thanthe brightness y(i, j) of the object pixel Pij. As a result, thecontrast improvement amount VRPiJ becomes a value smaller than 1.0,which is the value in the center. Therefore, the values of the enhancedimage obtained by the pixel value conversion unit 102 become low values.

By intentionally suppressing or increasing the amount corresponding tothe denominator when finding the contrast improvement amount in thisway, the values of the pixels obtained from this contrast improvementamount can be controlled. In this way, it is possible to suppressdecreases in the contrast improvement amount in uniformly largehighlighted areas or sudden increases in the contrast improvement amountin uniformly large shadow areas somewhat by a very simple process.

Therefore, as shown in FIG. 48, it is possible to output the enhancedimage obtained by the contrast improvement unit 11 as is as the outputimage without having an image combination unit 12 in the imageprocessing apparatus.

Of course, in order to more accurately suppress decreases in thecontrast improvement amount in highlighted areas or sudden increases inthe contrast improvement amount in shadow areas, it is possible to usean image combination unit 12 and to adaptively combine the input imageand enhanced image.

In this embodiment, and example was explained that used Equations 15 and16, however it is also possible to calculate the contrast improvementamount by using Equations 4 and 5. In this case, the upper and lowerlimits of the three components can be matched, however, in order thatthe balance of the finally obtained output image is not lost, it ispreferred that upper and lower limits be set for each of the components.

Moreover, as shown in FIG. 49, it is possible to combine theconstruction of the contrast improvement unit 11 of the image processingapparatus of this embodiment with the construction of the contrastimprovement apparatus 11 of the second embodiment.

Also, the image processing of this twelfth embodiment can be realized bya software process that uses a CPU and DSP.

Thirteenth Embodiment

As shown in FIG. 50, in addition to an image input unit 10, imagecombination unit 12 and image output unit 13, the image processingapparatus of a thirteenth embodiment of the invention comprises apre-processing unit 15 and a post-processing unit 16.

The pre-processing unit 15 performs pre-processing such as inverse gammaconversion of the image input from the image input unit 10. In recentyears, images taken by a digital still camera (DSC) have increase asimage data, which become the object of contrast improvement processing.When displaying the image data taken by a DSC on a monitor such as apersonal computer, in order to make the input image clearer, aconversion process using a gamma function G1 that has an upwardly convexshape as shown in FIG. 51 is normally performed. In a normally used sRGBspace, a gamma value of 2.2 is used for this gamma conversion. Even whenoutputting the image to printer, the image processing is executed foroutputting the image on the assumption that this gamma function has beenapplied beforehand to the input image. In order to accurately extractimage data obtained by a photographic element such as a CCD, it isnecessary to perform a conversion process using an inverse gammafunction to restore the effect of γ=2.2. In this case, thepre-processing unit 15 performs inverse gamma conversion using afunction G2 that has a downward face convex shape.

The contrast improvement unit 11 performs the contrast improvementprocess on the image from the pre-processing unit 15 instead of theimage from the image input unit 10.

In this embodiment, as shown in FIG. 52, in additions to the correctiondata calculation unit 100 and pixel value conversion unit 102, thecontrast improvement unit 11 comprises a comparison pixel setting unit115 and conversion standard value calculation unit 116, but does notcomprise an extraction unit 101.

As shown in FIG. 53, the comparison pixel setting unit 115 initializesthe position of the object pixel Pij, and selects comparison pixels fromamong the pixels in the area corresponding to the surround view of theobject pixel to be used in the calculation of the contrast improvementamount VRPij.

The correction data calculation unit 100 uses the values of thecomparison pixels to find the contrast improvement amount VRPij. Asshown in FIG. 54, the correction data calculation unit 100 comprises asurrounding average unit 150 and improvement amount calculation unit151.

The surrounding average unit 150 finds the average value of thecomparison pixels. The improvement amount calculation unit 151calculates the contrast improvement amount from the average value thatwas found and the value of the object pixel Pij. Here, Equations 15 and16 are used to calculate the contrast improvement amount instead ofEquations 4 and 5.

After the contrast improvement amount has been found for each pixel, theconversion standard value calculation unit 116 finds the conversionstandard value VTc (trc(i, j), tgc(i, j), tbc(i, j)) for converting thecontrast improvement amount VRPij (Rr(i, j), Rg(i, j), Rb(i, j)) to thevalue of the pixel of the enhanced image v.

The pixel value conversion unit 102 converts the contrast improvementamount VRPij to the value of the pixel of the enhanced image v based onthe found conversion standard value VTc.

In this embodiment, the contrast improvement amount VRPij that isnormalized in the range 0.0 to 1.0 is multiplied by the conversionstandard value VTc, to convert the data to real number data within therange 0.0 to 1.0. Also, this real number data is converted to integerdata that is expressed by the 8 bits 0 to 255, for example.

The conversion standard value VTc can be given by Equations 17, forexample, using the brightness y(i, j) of the object pixel Pij in thepre-processed input image vi.trc=Cen+Dev×(y(ij)−Ycen)trc=Cen+Dev×(y(ij)−Ycen)trc=Cen+Dev×(y(ij)−Ycen)  (Eqs. 17)

Here, Cen is the center value of the conversion standard value VTc, Devis the amount of displacement with respect to the brightness, and Ycenis the center value of the brightness. Here, VTc=0.5, Dev=0.2 andYcen=0.5 will be used. Ycen can also be the average brightness value.

The reason for giving a conversion standard value VTc such as that fromEquation 17 is to convert to a higher value the higher the brightnessis, and convert to a lower value the lower the brightness is. Theconversion standard values VTc can be uniformly set to the averagevalue, VTc=(0.5, 0.5, 0.5), however, by controlling the conversionstandard value VTc as described above, it is possible to suppress suddenchanges from the input image. It is also possible to suppress suddenincreases in brightness of shadow areas.

However, when the slope in Equations 17 becomes too large, contrastimprovement in dark areas is suppressed, and the original feature ofenhancing and improving the contrast in areas where there is adifference in change is lost.

Also, in this embodiment, values are converted to the desired pixelvalues after first converting them to real number data in the range 0.0to 1.0. Therefore, it is preferred that the conversion standard valueVTc be defined such that the obtained value for the conversion standardvalue VTc is in the range 0.0 to 1.0.

Furthermore, in this example, the conversion standard value VTc is givenusing Equations 17, however it is not limited to this. There is no needfor a linear relationship as in Equations 17, and it is also possible touse a non linear relationship. Here, a process of multiplying thecontrast improvement amount VRPij that is normalized in the range 0.0 to1.0 by the conversion standard value VTc was performed, however, it isalso possible to set the conversion standard value VTc such that takesvalues in the range 0 to 255 such as Cen=140, Dev=0.2, Ycen=128, withoutnormalizing the contrast improvement amount VRPij. When doing this, thebrightness y(i, j) is defined such that it takes a value in the range 0to 255, and a process of keeping the value obtained after multiplicationby the conversion standard value VTc within the range 0 to 255 isnecessary.

After the values of the enhanced image have been found for all pixels,the image combination unit 12 adaptively combines the obtained enhancedimage and the input image in the same way as was done in the first andninth embodiments, for example. Here, as shown in FIG. 55, the imagecombination unit 12 comprises a combination coefficient calculation unit201, weighted average combination unit 202 and output value setting unit203.

The combination coefficient w1 for the enhanced image is given byEquation 18 below using the thresholds TH and THComb, and thecombination coefficient w0 for the input image is given as w0=1.0−w1.w1(i,j)=exp(−y(i,j)² /TH ²)where zz=wy(i,j)/y(i,j)ww=THComb/(zz+THComb)if (w1(i,j)>ww)w1(i,j)=ww  (Eq. 18)

This Equation 18 gives priority of the values of the enhanced image overthe input image in dark areas. When the improvement is sudden based onthe ratio between the brightness of the input image and the brightnessof the enhanced image, the noise enhancement in dark areas is suppressedby suppressing the combination coefficient w1.

Also, w1(i, j) can be defined as in Equation 19 below.w1(i,j)=ww·exp(−y(i,j)² /TH ²)  (Eq. 19)

In this case, there is a possibility that the w1(i, j) of the enhancedimage in dark areas will be suppressed, however, since w1(i, j) changescontinuously, it is possible to suppress sudden change.

In the case of giving the combination coefficient using Equation 18,overhead in the processing time by the exponential function is possible,however, by putting the exponent function according to preset precisionin a table, the overhead due to that processing time can be suppressedby referencing that table.

As shown in FIG. 56, after obtaining a combined image using thecombination coefficients w0 and w1, the post-processing unit 16 performspost-processing on the combined image such as the gamma conversion thatwas applied originally to the input image.

After the post-processing unit 16 performs this kind of post-processing,the image output unit 13 generates the final output image to be outputto a device such as a printer or display from that image.

In the image processing of this embodiment, it is possible to easily andaccurately perform contrast improvement in which noise enhancement dueto decreases in brightness in highlighted areas and sudden increases inbrightness in shadow areas are suppressed. Furthermore, since theextraction process by the extraction unit 101 is not needed, it ispossible to reduce the amount of processing.

It is possible to combine the construction of the contrast improvementunit 11 of this embodiment with the construction of the contrastimprovement apparatus 11 of the second embodiment.

Moreover, the image processing of this thirteenth embodiment can also berealized by a software process that uses a CPU and DSP.

Fourteenth Embodiment

The image processing apparatus of a fourteenth embodiment of theinvention has the same construction as the image processing apparatus ofthe thirteenth embodiment, however, as shown in FIG. 57, in addition tothe surrounding average unit 150 and improvement amount calculation unit151, the correction data calculation unit 100 comprises an edge datadetection unit 152, correction coefficient calculation unit 153 andcomparison amount correction unit 154.

As shown in FIG. 58, in this image processing apparatus, after thepre-processed input image is input to the contrast improvement unit 11,the comparison pixel setting unit 115 of the correction data calculationunit 100 initializes the position of the object pixel Pij and thenselects comparison pixels.

The surrounding average unit 150 calculates the weighted average valueVAPij (Ar(i, j), Ag(i, j), Ab(i, j)) of the comparison pixels as in thethirteenth embodiment.

The edge data detection unit 152 finds edge data ed(i, j) of thebrightness y(i, j) of the object pixel Pij.

The correction coefficient calculation unit 153 finds a correctioncoefficient VDAPij (dAr(i, j), dAg(i, j), dAb(i, j)) for correcting theaverage value VAPij based on the edge data ed(i, j).

The comparison amount correction unit 154 corrects the average valueVAPij based on the correction coefficient VDAPij. Here, average valueVAPij is multiplied by the correction coefficient VDAPij for each of thecomponents. Also, the improvement amount calculation unit 151 uses thecorrected average value VAPij to find the contrast improvement amountVRPij.

The correction coefficient can be given according to Equation 20 usingthe average value ave_ed of the edge data ed(i, j) for each pixel of thepre-processed input image.dAr=dAg=dAb=dAcen−dAdev×(ed(i,j)−ave _(—) ed)Rr(i,j)=r(i,j)/(dAr×Ar(i,j))Rg(i,j)=g(i,j)/(dAg×Ag(i,j))Rb(i,j)=b(i,j)/(dAb×Ab(i,j))  [Eqs. 20]

Here, dAcen is the standard value of the correction coefficient, anddAdev is amount of change of the correction coefficient, where dAcen=1.0and dAdev=0.3, for example. Care must be taken that dAr, dAg and dAb donot become negative.

In this embodiment, a correction coefficient is used in this way tocontrol the average value VAPij of the surrounding. That is, near theedges, the correction coefficient becomes less than 1.0, and as a resultthe average value VAPij is affected by it and is reduced. Therefore, thecontrast improvement amount VRPij that is calculated using the ratio ofeach component of the object pixel with respect to this average valueVAPij becomes large near the edges, and by used the edge data other thanfor the average brightness difference with respect to the surroundings,it is possible to improve the sharpness in the combined image. On theother hand, in flat areas the correction coefficient becomes greaterthan 1.0, and as a result, the average value VAPij of the surroundingsincreases so the contrast improvement amount decreases.

In order to simplify the process, it is possible to set the averagevalue ave_ed in Equation 20 to 0.5. Furthermore, the equation used forcontrolling the average value VAPij is not limited to this. It is notnecessary to have a linear relationship as in Equation 20, and it ispossible to have a non linear relationship.

Flatness is removed by controlling the average value VAPij, so it isalso possible to set the conversion standard value VTc to a uniformcenter value Cen without being controlled. In the case where theconversion standard value VTc is controlled together with controllingthe average value VAPij, there is a possibility of excessive enhancementnear the edges, so increases in the correction coefficient of theaverage value VAPij is suppressed according to increases in thebrightness by making dAdev smaller according to the brightness or thelike.

In this way, the contrast in dark areas is improved while at the sametime the sharpness of the combined image is somewhat preserved.

It is possible to combine the image processing of this embodiment withthe image processing of the second embodiment.

Moreover, the image processing of this fourteenth embodiment can also berealized by a software process that uses a CPU and DSP.

Furthermore, the average value VAPij was controlled here by the edgeamount, however, it is also possible to multiply the contrastimprovement amount VRPij by a correction coefficient ttt that is definedby the edge amount as in Equations 21 below. Here, tcen can be set to0.5 and tdev can be set to 0.3.ttt=tcen+tdev×(ed(i,j)−ave _(—) ed)Rr′(i,j)=ttt×Rr(i,j)Rg′(i,j)=ttt×Rg(i,j)Rb′(i,j)=ttt×Rb(i,j)  [Eqs. 21]

Fifteenth Embodiment

The construction of the correction data calculation unit 100 of theimage processing apparatus of a fifteenth embodiment of the invention ispartially different than that of the image processing apparatus of thethirteenth embodiment. As shown in FIG. 59, in addition to thesurrounding average unit 150 and improvement amount calculation unit151, the correction data calculation unit 100 comprises a brightnessdifference average unit 155, enhancement component calculation unit 156and improvement amount correction unit 157.

As shown in FIG. 60, in this image processing image, when the inputimage before processing is input to the contrast improvement unit 11,the comparison pixel setting unit 115 selects a comparison pixel fromamong the pixels in the area corresponding to the surround view of theobject pixels Pij of the input image that will be used for calculatingthe contrast improvement amount.

After comparison pixels have been selected by the comparison pixelsetting unit 16, the surrounding average unit 150 of the correction datacalculation unit 100 calculates the weighted average value VAPij for thevalue of the comparison pixel in the same way as was done in thethirteenth embodiment, and the improvement amount calculation unit 151finds the contrast improvement amount VRPij.

Also, the brightness difference average unit 155 finds the weighted sumAdy(i, j) of the absolute value of the difference between the brightnessy(i, j) of the object pixel Pij and the brightness ys(i, j) of thecomparison pixel Q(i, j).

The enhancement component calculation unit 156 calculates theenhancement components VBRPij (BRr(i, j), BRg(i, j), BRb(i, j)) forenhancing the contrast improvement amount VRPij based on this value Ady(i, j).

The improvement amount correction unit 157 corrects the contrastimprovement amount VRPij based on the calculated enhancement componentsVBRPij.

The enhancement components VBRPij can be expressed by the followingequations 22.

$\quad\begin{matrix}{{{{{Ady}\left( {i,j} \right)} = {\sum\limits_{u}^{\;}\;{\sum\limits_{v}^{\;}\;{{{{{y\left( {i,j} \right)} - {{ys}\left( {u,v} \right)}}}/{{len}\left( {u,v} \right)}}/C^{2}}}}}{{{BRr}\left( {i,j} \right)} = {\left( {{{Rr}\left( {i,j} \right)} - 1.0} \right) \times {CVal} \times {{Ady}\left( {i,j} \right)}}}{{{BRg}\left( {i,j} \right)} = {\left( {{{Rg}\left( {i,j} \right)} - 1.0} \right) \times {CVal} \times {{Ady}\left( {i,j} \right)}}}{{BRb}\left( {i,j} \right)} = {\left( {{{Rb}\left( {i,j} \right)} - 1.0} \right) \times {CVal} \times {{Ady}\left( {i,j} \right)}}}{{where},\;{{{len}\left( {u,v} \right)} = {{sqrt}\left( {{\left( {u - i} \right) \times \left( {u - i} \right)} + {\left( {v - j} \right) \times \left( {v - j} \right)}} \right)}}}} & \left\lbrack {{Eqs}.\mspace{14mu} 22} \right\rbrack\end{matrix}$

In equations 22, CVa1 is a positive constant, and controls the amount ofedge enhancement.

BRr(i, j), BRg(i, j) and BRb(i, j) can also be defined by the followingexpressions 23 such that they are further controlled by the brightnessof the object pixel. Here, C′val is a positive constant.BRr(i,j)=(Rr(ij)−1.0)×kkk×C′valBRg(i,j)=(Rg(ij)−1.0)×kkk×C′valBRb(i,j)=(Rb(ij)−1.0)×kkk×C′valkkk=y(ij)×Ady(ij)  [Eqs. 23]

In Equations 22 the value Ady(i, j) is taken to be the amount ofdisplacement enhancement of the object pixel Pij, and Equations 22calculate the component amounts for performing enhancement of the amountof displacement from 1.0 of the contrast improvement amount VRPij (Rr(i,j), Rg(i, j), Rb(i, j)) using the value Ady(i, j) and a presetenhancement coefficient.

For example, there is no enhancement component for places where there isno change in uniform brightness. However, in the case of an object pixelwhose contrast improvement amount is greater than 1.0, components arecalculated such as to further enhance the pixel values of the enhancedimage. On the other hand, in the case of an object pixel whose contrastimprovement amount is less than 1.0, a component is obtained that willsuppress the pixel value of the object pixel. Therefore, in the enhancedimage, the sharpness in areas where the change in the pixel values islarge such as near the edges increases, and it is possible to save thesharpness of the finally obtained combined image.

As described above, in the image processing of this embodiment, it ispossible to perform contrast improvement in dark areas such thatsharpness can be saved in the combined image.

It is possible to combine the image processing of this embodiment withthe image processing in the second embodiment.

Moreover, the image processing of this fifteenth embodiment can also berealized by a software process that uses a CPU and DSP.

Sixteenth Embodiment

The construction of the correction data calculation unit 100 of theimage processing apparatus of a sixteenth embodiment of the invention ispartially different than that of the image processing apparatus of thethirteenth embodiment. As shown in FIG. 61, instead of the correctiondata calculation unit 100, the contrast improvement unit 11 comprises avertical direction addition unit 117, a simple surrounding average unit118, a comparison amount correction unit 119 and an improvement amountcalculation unit 151.

Furthermore, the comparison pixel setting unit 115 removes pixels in thevertical direction at a certain interval and sets a comparison pixelfrom among the pixels of a rectangular area corresponding to thesurround view of the object pixel Pij. In the example shown in FIG. 62,of the pixels of a 7 pixel×7 pixel rectangular area Mi, the even numberpixels in the vertical direction starting from the top are removed, andthe other pixels are selected. The number of selected pixels m is 28.The size of the rectangular area and the interval for removing pixelsare not limited to those described above.

The vertical direction addition unit 117 finds the addition values N[i](here, i=0, 1, . . . , 6) in the vertical direction of the brightness ofthe comparison pixel at each horizontal pixel position i of therectangular area.

The simple surrounding average unit 118 finds the added sum S of theaddition values N[k] for each horizontal pixel position of therectangular area. Also, the average comparison object brightness Asy(i,j) is calculated by dividing the added total S by the number ofcomparison pixels m.

The improvement amount calculation unit 151 finds the contrastimprovement amount VRPij by comparing this average comparison objectbrightness Asy(i, j) with the values VPij (r(i, j), g(i, j), b(i, j)) ofthe object pixels Pij.

In this way, the contrast improvement amount VRPij for the object pixelPij is found, and when the horizontal pixel position of the object pixelPij moves from position i to position i+1, the surrounding rectangulararea also moves from area Mi to area Mi+1.

When the object pixel moves, the vertical direction addition unit 117finds the addition value N for a comparison pixel at the end in thehorizontal direction from among the pixels of the new area Mi+1. In theexample shown in FIG. 62, the addition value N[7] is calculated.

The simple surrounding average unit 118 subtracts the addition value Nof a comparison pixel, which is at the start in the horizontal directionfrom among the pixels of the area Mi+1, from the weighted sum S at theprevious position. In the example shown in FIG. 62, the addition valueN[0] is subtracted from the weighted sum S. Also, by adding the newlycalculated addition value N[7] to that result, the sum S of thebrightness is obtained for the area Mi+1.

By dividing this value by the number of comparison pixels m in the sameway, the average comparison object brightness Asy(i+1, j) is obtained.

By adding and subtracting the pre prepared addition values N[k] in thevertical direction in this way, it is possible to easily obtain theaverage comparison object brightness.

As described above, in the image processing of this embodiment, bydesigning and simplifying the calculation of the filtering process,which has the largest amount of processing, it is possible to save theedges and improve the contrast while at the same time reduce theprocessing time.

It is also possible to combine the image processing of this embodimentwith the image processing of the second embodiment. In the case ofperforming multi-resolution processing as in the second embodiment, theamount of processing becomes very large, however, by designing thecalculation and calculation procedure for calculating the weighted sumas in this embodiment, it is possible to greatly improve the problem ofa large amount of processing.

Also, in the image processing of this embodiment, it is possible to addthe control of the average comparison density VAPij of the fourteenthembodiment, and the control of the enhancement component VBRPij of thefifteenth embodiment.

Moreover, the image processing of this sixteenth embodiment can also berealized by a software process that uses a CPU and DSP.

Seventeenth Embodiment

The construction of the contrast improvement unit 11 of the imageprocessing apparatus of a seventeenth embodiment of the invention ispartially different than that of the image processing apparatus of thesixteenth embodiment. As shown in FIG. 63, instead of the verticaldirection addition unit 117, the contrast improvement unit 11 comprisesa removal vertical direction addition unit 120.

Furthermore, the comparison pixel setting unit 115 removes pixels in thevertical direction and horizontal direction at a certain interval fromthe pixels of a rectangular area that corresponds to the surround viewof the object pixel Pij, and sets a comparison pixel Qij. In the exampleshown in FIG. 64, of the pixels of a 7 pixel×7 pixel rectangular areaMi, the even number pixels in the vertical direction and horizontaldirection starting from the first are removed, and the other pixels areselected. The size of the rectangular area and the interval for removingpixels in the vertical and horizontal directions are not limited tothose described above.

The removal vertical direction addition unit 120 finds addition valuesN[h] (here, h=0, 2, 4, 6) in the vertical direction of the brightness ofthe comparison pixels located at odd numbered positions counting fromthe first from among each of the horizontal pixel positions i in therectangular area.

The addition values N[h+1] in the vertical direction of the brightnessof the comparison pixels at even numbered positions from the start ofthe horizontal pixel positions i are given by N[h+1]=N[h]. For example,a value that is the same as addition value N[0] is used for additionvalue N[1], and a value that is the same as addition value N[2] is usedfor addition value N[3].

The simple surrounding average unit 118 finds the weighted sum S of theaddition value N[k] for each horizontal pixel position in therectangular area. Also, as in the sixteenth embodiment, the averagecomparison object brightness Asy(i, j) is calculated by dividing theweighted sum S by the number of comparison pixels m.

The improvement amount calculation unit 151 finds the contrastimprovement amount VRPij by comparing this average comparison objectbrightness Asy(i, j) with the values VPij (r(i, j), g(i, j), b(i, j)) ofthe object pixel Pij.

When the horizontal pixel position of the object pixel Pij moves fromposition i to position i+1, the removal vertical direction addition unit120 finds the addition values N for the comparison pixels at the veryend in the horizontal direction from among the pixels of thecorresponding area Mi+1. However, when that position is in a evennumbered position counted from the start, the pixel in that position isa pixel to be removed and is not selected, so the addition values N isnot found newly. In the example shown in FIG. 64, addition value N[7] isnot calculated, and is interpolated as a value that is the same asaddition value N[6].

Of the pixels of the area Mi+1, the simple surrounding average unit 118subtracts the addition value N for the comparison pixel at the verystart in the horizontal direction from the weighted sum S at theprevious positions. In the example shown in FIG. 64, the additions valueN[0] is subtracted from the weighted sum S. Also, by adding additionvalue N[7] to that result, the sum S of the brightness for area Mi+1 isobtained.

The average comparison object brightness Asy(i+1, j) is obtained bydividing this value in the same way by the number of comparison pixelsm.

As described above, in the image processing of this embodiment, bydesigning and simplifying the calculation of the filtering process,which has the largest amount of processing, it is possible to save theedges and improve the contrast while at the same time reduce theprocessing time.

It is also possible to combine the image processing of this embodimentwith the image processing of the second embodiment. In the case ofperforming multi-resolution processing as in the second embodiment, theamount of processing becomes very large, however, by designing thecalculation and calculation procedure for calculating the weighted sumas in this embodiment, it is possible to greatly improve the problem ofa large amount of processing.

Also, in the image processing of this embodiment, it is possible to addthe control of the average comparison density VAPij of the fourteenthembodiment, and the control of the enhancement component VBRPij of thefifteenth embodiment.

Moreover, the image processing of this seventeenth embodiment can alsobe realized by a software process that uses a CPU and DSP.

Eighteenth Embodiment

In the thirteenth embodiment described above, the contrast improvementamount VRPij (Rr(i, j), Rg(i, j), Rb(i, j)) was calculated for eachcomponent by Equations 15 and 16. In this eighteenth embodiment, thecontrast improvement amount yRP(i, j) for the brightness is calculatedas using Equations 22 instead of Equations 16, and using the sameprocedure as in the thirteenth embodiment as shown in FIG. 65.

The improved brightness yRP(i, j) is calculated from the ratio of thebrightness y(i, j) of the object pixel Pij and the average Ay(i, j) ofthe brightness of the pixels of the area corresponding to the surroundview. Also, here, the conversion standard value calculation unit 116finds the standard value yTc(i, j) for converting the improvedbrightness yRP(i, j) to the brightness of the pixel on the enhancedimage v. The pixel value conversion unit 102 converts the improvedbrightness yRP(i, j) to the brightness of the pixel on the enhancedimage v based on the found standard value yTc(i, j).

Furthermore, in order to combine the enhanced image v and the inputimage vi, the image combination unit 12 calculates the brightness of thepixels of both images instead of calculating the values for eachcomponent of the pixels of both images.

In this embodiment, the process of combining images is performed usingjust the brightness in this way. The reason for not performing theprocess for each component as in the case of the thirteenth embodimentis in order to further maintain the color balance of the input image inthe enhanced image. As shown in FIG. 66A, when processing is performedfor each component as in the thirteenth embodiment, there is apossibility that the color balance is lost in the enhancement processand correction process, so a change in color may occur between the inputimage and combined image. On the other hand, by finding the improvedcolor differences wcr(i, j) and wcb(i, j) by reflecting the ratioRatio(i, j) between the brightness wy(i, j) of the combined image andbrightness y(i, j) on the color differences cr(i, j) and cb(i, j) of theinput image, it is possible to maintain the color balance of the inputimage in the combined image, as shown in FIG. 66B.

In this embodiment, a post-processing unit 16 finds the improved colordifferences wcr(i, j) and wcb(i, j) and finds the value of the pixel ofthe combined image. For doing this, as shown in FIG. 67, in addition tohaving the same construction as the image processing apparatus of thethirteenth embodiment, the post-processing unit 16 of this imageprocessing apparatus further comprises an input brightness/colordifference calculation unit 160, a brightness ratio calculation unit161, a brightness adjustment unit 162, a color difference componentcorrection unit 163, image regeneration unit 164 and gamma conversionunit 165.

After image combination unit 12 generates a combined image for thebrightness, the input brightness/color difference calculation unit 160of the post-processing unit 16 initializes the position of the objectpixel Pij and then calculates the brightness y(i, j) and colordifferences cr(i, j) and cb(i, j) of the object pixel of the inputimage, as shown in FIG. 68.

The brightness ratio calculation unit 161 compares the brightness wy(i,J) of the object pixel Pij in the combined image with the brightnessy(i, j) of the corresponding pixel in the input image, and calculatesthe ratio Ratio(i, j) between them.

The brightness adjustment unit 162 adjusts the brightness wy(i, j) ofthe combined image based on that brightness ratio. By just multiplyingthe color differences cr(i, j) and cb(i, j) of the input image by thebrightness ratio, there is a possibility that the color differenceswcr(i, J) and wcb(i, j) of the combined image may be saturated.Therefore, the brightness adjustment unit 162 determines whether or notthe color difference wcr(i, j) or wcb(i, j) of the combined imageobtained by multiplication became saturated. In the case where the colordifference wcr(i, j) becomes saturated, it corrects the brightness ratioRatio(i, j) by the ratio with that saturated color difference wcr(i,j)/cr(i, j). The improved brightness dwy(i, j) after combination iscalculated based on that value.

The color difference component correction unit 163 corrects the colordifferences cr(i, j) and cb(i, j) of the input image based on thecorrected ratio Ratio(i, j) between the brightness wy(i, j) andbrightness y(i, j), and generates the color differences aftercombination dwcr(i, j) and dwcb(i, j).

The image regeneration unit 163 calculates the value VWPij (wr(i, j),wg(i, j), wb(i, j)) of the pixel in the combined image based on thecorrected color difference components and brightness.

After the values VWPij of the pixels in the combined image are found,the gamma conversion unit 164 performs gamma conversion processing,which was originally performed on the input image, on the combinedimage. By doing so, the output image that is output from the imageoutput unit 13 is generated.

In the image processing of this embodiment, in order to keep the colorbalance of the input image, processing is only performed on thebrightness when performing the contrast improvement process. Also, bycorrecting the color difference of the input image based on theimprovement ratio, it is possible to improve the color balance of thecombined image. By taking into consideration the saturation state of thecolor difference after correction, it is possible to perform contrastimprovement while maintaining the color balance of the input image.Therefore, it is not necessary to separately have a means or method forimproving the color balance.

Moreover, the image processing of this eighteenth embodiment can also berealized by a software process that uses a CPU and DSP.

1. An image processing apparatus comprising: a contrast improvement unitoperable to perform a contrast improvement process on an input image bycomparing an object pixel of the input image with pixels that belong tosurrounding areas of the object pixel, the contrast improvement unitcomprising a correction data calculation unit operable to find acontrast improvement amount for a pixel in the input image, anextraction unit operable to extract an effective range from distributionof the contrast improvement amount, and a pixel value conversion unitoperable to convert a contrast improvement amount of the object pixel toa value of a corresponding pixel in the enhanced image according to theextracted range; an image combination unit operable to combine anenhanced image obtained by the contrast improvement unit and the inputimage; and an image output unit operable to output the image aftercombination, wherein the pixel value conversion unit comprises astandard intensity calculation unit operable to calculate a standardintensity value that indicates contrast intensity of the input image, aconversion curve estimation unit operable to estimate a conversion curvefor converting the contrast improvement amount to a value in theenhanced image based on the standard intensity value, and a pixel valueestimation unit operable to use the conversion curve to convert thecontrast improvement amount to a value in the enhanced image, andwherein the conversion curve estimation unit comprises an initialcandidate setting unit operable to set an initial population of searchvectors that indicate conversion curves, a pixel value conversioncandidate calculation unit operable to find a conversion value in acandidate for an enhanced image from the contrast improvement amountusing a conversion curve candidate corresponding to each search vector,an evaluation value calculation unit operable to use the standardintensity value and the conversion value to calculate an evaluationvalue for evaluating the candidates of each conversion curve, a fitnesscalculation unit operable to calculate fitness of a candidate of eachconversion curve based on the evaluation value, a recombinationoperation unit operable to perform recombination operation on a searchvector selected based on the fitness of the candidates of eachconversion curve and generate a next generation population, and anestimation end judgment unit operable to determine whether or notestimation of the conversion curve ends at a current generation.
 2. Animage processing apparatus comprising: a contrast improvement unitoperable to perform a contrast improvement process on an input image bycomparing an object pixel of the input image with pixels that belong tosurrounding areas of the object pixel, the contrast improvement unitcomprising a signal conversion unit operable to convert a value of pixelin the input image to a plurality of signals that include a signal thatis an object of contrast improvement, an object correction datacalculation unit operable to find a contrast improvement amount of theobject pixel for an object signal obtained from the signal conversionunit, an extraction unit operable to extract an effective range fromdistribution of the contrast improvement amount for the object signal,an object signal conversion unit operable to convert the contrastimprovement amount for the object signal to a value of the object signalin the enhanced image, and a signal inverse conversion unit operable tofind a value of pixel in the enhanced image based on the object signalof the enhanced image and signals other than the object signal obtainedby the signal conversion unit; an image combination unit operable tocombine an enhanced image obtained by the contrast improvement unit andthe input image; and an image output unit operable to output the imageafter combination, wherein the object signal conversion unit comprises astandard intensity calculation unit operable to calculate a standardintensity value that indicates contrast intensity of the input image forthe object signal obtained by the signal conversion unit, an objectsignal conversion curve estimation unit operable to estimate aconversion curve for converting the contrast improvement amount for theobject signal to the value in the enhanced image based on the standardintensity value, and an object signal estimation unit operable to usethe estimated conversion curve to convert the contrast improvementamount for the object signal to the value in the enhanced image, andwherein the object signal conversion curve estimation unit comprises aninitial candidate setting unit operable to set an initial population ofsearch vectors that indicate conversion curves, an object signalconversion candidate calculation unit operable to find a conversionvalue for the object signal in a candidate for an enhanced image fromthe contrast improvement amount for the object signal using a conversioncurve candidate corresponding to each search vector, an evaluation valuecalculation unit operable to use the standard intensity value and theconversion value to calculate an evaluation value for evaluating thecandidates of each conversion curve, a fitness calculation unit operableto calculate fitness of a candidate of each conversion curve based onthe evaluation value, a recombination operation unit operable to performrecombination operation on a search vector selected based on the fitnessof the candidates of each conversion curve and generate a nextgeneration population, and an estimation end judgment unit operable todetermine whether or not estimation of the conversion curve ends atcurrent generation.
 3. An image processing apparatus comprising: acontrast improvement unit operable to perform a contrast improvementprocess on an input image by comparing an object pixel in the inputimage with pixels that belong to surrounding areas of the object pixel;a density correction unit operable to correct density distribution of anenhanced image obtained by the contrast improvement unit according todensity distribution of the input image; an image combination unitoperable to combine a corrected image obtained by the density correctionunit and the input image; and an image output unit operable to output animage after combination, wherein the contrast improvement unit comprisesa comparison pixel setting unit operable to set positions in a verticaldirection of pixels to be used in the comparison from among pixels inthe area surrounding the object pixel, a vertical direction additionunit operable to add weighting in the vertical direction to the densityof the comparison pixels obtained by the comparison pixel setting unit,a simple surrounding average unit operable to calculate comparisondensity for the object pixel from a value at each horizontal pixelposition in the surrounding area obtained by the vertical directionaddition unit, an improvement amount calculation unit operable to find acontrast improvement amount from the comparison density and density ofthe object pixel a conversion standard value calculation unit operableto find a conversion standard value for converting the contrastimprovement amount to a value of pixel in the enhanced image, and apixel value conversion unit operable to convert the contrast improvementamount to a value of pixel in the enhanced image based on the conversionstandard value.
 4. An image processing apparatus comprising: apre-processing unit operable to perform pre-processing on an inputimage; a contrast improvement unit operable to perform a contrastimprovement process on the pre-processed image by comparing an objectpixel of the pre-processed image with pixels that belong to surroundingareas of the object pixel; an image combination unit operable to combinean enhanced image obtained by the contrast improvement unit and theinput image; a post-processing unit operable to perform post-processingon an image after combination; and an image output unit operable tooutput the post-processed image, wherein the contrast improvement unitcomprises a comparison pixel setting unit operable to set positions in avertical direction of pixels to be used in the comparison from amongpixels in the area surrounding the object pixel, a vertical directionaddition unit operable to add weighting in the vertical direction to thedensity of the comparison pixels obtained by the comparison pixelsetting unit, a simple surrounding average unit operable to calculatecomparison density for the object pixel from a value at each horizontalpixel position in the surrounding area obtained by the verticaldirection addition unit, an improvement amount calculation unit operableto find a contrast improvement amount from the comparison density anddensity of the object pixel, a conversion standard value calculationunit operable to find a conversion standard value for converting thecontrast improvement amount to a value of pixel in the enhanced image,and a pixel value conversion unit operable to convert the contrastimprovement amount to a value of pixel in the enhanced image based onthe conversion standard value.
 5. An image processing apparatuscomprising: a pre-processing unit operable to perform pre-processing onan input image; a contrast improvement unit operable to perform acontrast improvement process on the pre-processed image by comparing anobject pixel of the pre-processed image with pixels that belong tosurrounding areas of the object pixel; an image combination unitoperable to combine an enhanced image obtained by the contrastimprovement unit and the input image; a post-processing unit operable toperform post-processing on an image after combination; and an imageoutput unit operable to output the post-processed image, wherein thecontrast improvement unit comprises a comparison pixel setting unitoperable to set positions in a vertical direction and positions in ahorizontal direction of pixels to be used in the comparison from amongpixels in the area surrounding the object pixel, a removal verticaldirection addition unit operable to add weighting to the verticaldirection of the density of the comparison pixels obtained by thecomparison pixel setting unit, a simple surrounding average unitoperable to calculate comparison density for the object pixel based onan addition value obtained by the removal vertical direction additionunit, an improvement amount calculation unit operable to find a contrastimprovement amount from the comparison density and density of the objectpixel, a conversion standard value calculation unit operable to find aconversion standard value for converting the contrast improvement amountto a value of pixel in the enhanced image, and a pixel value conversionunit operable to convert the contrast improvement amount to a value ofpixel in the enhanced image based on the conversion standard value. 6.An image processing method comprising: a contrast improvement step ofperforming a contrast improvement process on an input image by comparingan object pixel of the input image with pixels that belong tosurrounding areas of the object pixel, the contrast improvement stepcomprising a step of finding a contrast improvement amount for a pixelin the input image, a step of extracting an effective range fromdistribution of the contrast improvement amount, and a pixel valueconversion step of converting the contrast improvement amount of theobject pixel to a value of corresponding pixel in the enhanced imageaccording to the extracted range; an image combination step of combiningan enhanced image obtained from the contrast improvement unit and theinput image; and a step of outputting an image after combination,wherein the pixel value conversion step comprises a step of calculatingthe standard intensity value that indicates contrast intensity of theinput image, a conversion curve estimation step of estimating aconversion curve for converting the contrast improvement amount to avalue in the enhanced image, and a step of using the conversion curve toconvert the contrast improvement amount to a value in the enhancedimage, and wherein the conversion curve estimation step comprises a stepof setting an initial population of search vectors that indicateconversion curves, a step of finding a conversion value in a candidatefor an enhanced image from the contrast improvement amount using aconversion curve candidate corresponding to each search vector, a stepof using the standard intensity value and the conversion value tocalculate an evaluation value for evaluating the candidates of eachconversion curve; a step of calculating fitness of a candidate of eachconversion curve based on the evaluation value, a step of performingrecombination operation on a search vector selected based on the fitnessof the candidates of each conversion curve and generating a nextgeneration population, and a step of determining whether or notestimation of the conversion curve ends at current generation.
 7. Animage processing method comprising: a contrast improvement step ofperforming a contrast improvement process on an input image by comparingan object pixel of the input image with pixels that belong tosurrounding areas of the object pixel; an image combination step ofcombining an enhanced image obtained from the contrast improvement unitand the input image; and a step of outputting an image after combinationwherein the contrast improvement step comprises a signal conversion stepof converting a value of pixel in the input image to a plurality ofsignals that include the signal that is an object of contrastimprovement, a step of finding a contrast improvement amount of theobject pixel for an object signal obtained from the signal conversionstep, a step of extracting an effective range from distribution of thecontrast improvement amount for the object signal, an object signalconversion step of converting the contrast improvement amount for theobject signal to a value of the object signal in the enhanced image, anda step of finding a value of pixel in the enhanced image based on theobject signal of the enhanced image and signals other than the objectsignal obtained by the signal conversion step, and wherein the objectsignal conversion step comprises a step of calculating a standardintensity value that indicates contrast intensity of the input image forthe object signal obtained by the signal conversion step, an objectsignal conversion curve estimation step of estimating a conversion curvefor converting the contrast improvement amount for the object signal toa value in the enhanced image, and a step of using the estimatedconversion curve to convert the contrast improvement amount for theobject signal to a value in the enhanced image, and wherein the objectsignal conversion curve estimation step comprises a step of setting aninitial population of search vectors that indicate conversion curves, astep of finding conversion value for the object signal in a candidatefor an enhanced image from the contrast improvement amount for theobject signal using a conversion curve candidate corresponding to eachsearch vector, a step of using the standard intensity value and theconversion value to calculate an evaluation value for evaluating thecandidates of each conversion curve, a step of calculating fitness ofthe candidates of each conversion curve based on the evaluation value, astep of performing recombination operation on a search vector selectedbased on the fitness of the candidates of each conversion curve andgenerating a next generation population, and a step of determiningwhether or not estimation of the conversion curve ends at currentgeneration.
 8. An image processing method comprising: a pre-processingstep of performing pre-processing on an input image; a contrastimprovement step of performing a contrast improvement process on thepre-processed image by comparing an object pixel of the pre-processedimage with pixels that belong to surrounding areas of the object pixel;a step of combining an enhanced image obtained by the contrastimprovement step and the input image; a post-processing step ofperforming post-processing on an image after combination; and a step ofoutputting the post-processed image, wherein the contrast improvementstep comprises a step of setting positions in a vertical direction ofpixels to be used in the comparison from among pixels in the areasurrounding the object pixel, a vertical direction addition step ofadding weighting in the vertical direction to the density of thecomparison pixels obtained by the setting a step of calculatingcomparison density for the object pixel from a value at each horizontalpixel position in the surrounding area obtained by the verticaldirection addition step, a step of finding a contrast improvement amountfrom the comparison density and density of the object pixel, a step offinding a conversion standard value for converting the contrastimprovement amount to a value of pixel in the enhanced image, and a stepof converting the contrast improvement amount to a value of the pixel inthe enhanced image based on the conversion standard value.
 9. An imageprocessing method comprising: a pre-processing step of performingpre-processing on an input image; a contrast improvement step ofperforming a contrast improvement process on the pre-processed image bycomparing an object pixel of the pre-processed image with pixels thatbelong to surrounding areas of the object pixel; a step of combining anenhanced image obtained by the contrast improvement step and the inputimage; a post-processing step of performing post-processing on an imageafter combination; and a step of outputting the post-processed image,wherein the contrast improvement step comprises a step of settingpositions in a vertical direction and positions in a horizontaldirection of pixels to be used in the comparison from among pixels inthe area surrounding the object pixel, a removal vertical directionaddition step of adding weighting to the vertical direction of thedensity of the comparison pixels obtained by the setting, a step ofcalculating comparison density for the object pixel based on an additionvalue obtained by the removal vertical direction addition step, a stepof finding a contrast improvement amount from the comparison density anddensity of the object pixel, a step of finding a conversion standardvalue for converting the contrast improvement amount to a value of pixelin the enhanced image, and a step of converting the contrast improvementamount to a value of pixel in the enhanced image based on the conversionstandard value.
 10. An image processing apparatus comprising: a contrastimprovement unit operable to perform a contrast improvement process onan input image by comparing an object pixel in the input image withpixels that belong to surrounding areas of the object pixel; a densitycorrection unit operable to correct density distribution of an enhancedimage obtained by the contrast improvement unit according to densitydistribution of the input image; an image combination unit operable tocombine a corrected image obtained by the density correction unit andthe input image; and an image output unit operable to output an imageafter combination, wherein the contrast improvement unit comprises acomparison pixel setting unit operable to set positions in a verticaldirection and positions in a horizontal direction of pixels to be usedin the comparison from among pixels in the area surrounding the objectpixel, a removal vertical direction addition unit operable to addweighting to the vertical direction of the density of the comparisonpixels obtained by the comparison pixel setting unit, a simplesurrounding average unit operable to calculate comparison density forthe object pixel based on an addition value obtained by the removalvertical direction addition unit, an improvement amount calculation unitoperable to find a contrast improvement amount from the comparisondensity and density of the object pixel, a conversion standard valuecalculation unit operable to find a conversion standard value forconverting the contrast improvement amount to a value of pixel in theenhanced image, and a pixel value conversion unit operable to convertthe contrast improvement amount to a value of pixel in the enhancedimage based on the conversion standard value.